Building an AI Knowledge Layer for My Portfolio

I wanted to explore what it would take to make the content of my portfolio usable as knowledge for an AI system.

The goal was not to build a complete AI assistant. Instead, I wanted to understand and build the infrastructure behind one: taking structured portfolio content, transferring it into a dedicated backend, transforming it into AI-ready knowledge, storing that knowledge as embeddings, and making it available through semantic retrieval.

The result is a separate AI backend that connects to the existing Content System of my portfolio.

The problem

My portfolio already has a structured content system built around MDX. Content is created, reviewed, validated, previewed, and published through a controlled workflow.

That creates a useful source of structured knowledge, but an AI system needs a different representation of that content.

I wanted a clear boundary between the two systems:

• The portfolio remains the source of truth for published content.
• The AI backend consumes published content through an explicit handoff.
• The AI backend is responsible for transforming and indexing that content.
• Retrieval can then expose relevant knowledge without coupling the portfolio directly to the AI infrastructure.

This separation also makes it possible to evolve the AI layer independently from the portfolio application.

Architecture

Content System
     ↓
Published MDX
     ↓
Content Handoff
     ↓
AI Backend
     ↓
Content Normalization
     ↓
Chunking
     ↓
Embeddings
     ↓
PostgreSQL + pgvector
     ↓
Semantic Retrieval API
     ↓
[ Next: LLM + Interface ]

The important architectural decision was to stop the knowledge layer at retrieval.

An LLM and a user-facing interface can be added later without making the knowledge backend responsible for the entire AI experience.

From published content to AI-ready knowledge

The AI pipeline starts with published portfolio content.

Drafts are deliberately excluded. The published MDX content remains the source of truth, while the AI backend maintains its own representation of that content for retrieval.

The handoff is explicit rather than tightly coupling the two applications.

The content payload carries information such as:

• stable content identifier
• collection
• content type
• title
• slug
• source path
• content
• editorial metadata

A stable identifier was important because filenames, slugs, titles, and collection locations can change over time. The AI backend should still be able to recognize the same underlying piece of content.

Content ingestion

I built an ingestion API in the AI backend to receive published content.

The ingestion process includes:

1. Receiving the content payload.
2. Validating the request.
3. Normalizing the content.
4. Creating or updating the corresponding source.
5. Splitting the content into chunks.
6. Generating embeddings.
7. Persisting the source, chunks, and embeddings.

Only published content is ingested. Drafts remain outside the AI pipeline.

Chunking

Content is divided into smaller chunks so that individual sections can be represented and retrieved independently.

Embeddings

Each chunk is converted into a vector embedding using an embedding model and stored alongside the original content.

Vector storage

The embeddings are stored in PostgreSQL using pgvector.

This gives the AI backend a persistent knowledge store that can be queried using vector similarity.

Semantic retrieval

The backend exposes a retrieval API that converts a query into an embedding and searches the stored vectors for relevant content.

The result is a set of content chunks that can later be provided to an LLM as context.

Live API

The backend is deployed on Railway and exposes its API publicly.

Current state

Done

• Content System with structured MDX content
• Stable content identifiers
• Published-content handoff
• AI ingestion API
• Content normalization
• Chunking
• Embedding generation
• PostgreSQL
• pgvector
• Vector persistence
• Semantic retrieval API
• Railway deployment

Next

The next layer is intentionally left separate from this backend:

• Connect retrieved context to an LLM
• Generate grounded responses
• Build the user-facing AI interface

Why I stopped here

The retrieval layer already provides the foundation required for an AI knowledge system.

Rather than turning this backend into a large platform, I decided to treat the LLM and interface as a separate project.

This keeps the knowledge layer focused on one responsibility: turning published content into searchable AI-ready knowledge.