Retrieval augmented generation sounds complicated and is actually simple in outline. You give a language model access to your documents by retrieving the relevant ones and pasting them into the prompt. The model answers using that context instead of relying only on what it memorized during training. The outline fits in a sentence. The reason RAG systems fail is never the outline. It is the details, and this post is about the details, because I have watched the same mistakes sink the same projects more than once.
This assumes you understand embeddings and similarity search already. If not, read vector databases explained first, because the entire retrieval step depends on those ideas.
The pipeline at a glance
A RAG system has two phases. There is an offline ingestion phase where you process your documents and store them, and an online query phase where you answer a user’s question. Ingestion looks like this: take your documents, split them into chunks, generate an embedding for each chunk, and store the vectors with their text and metadata in a vector database. Querying looks like this: embed the user’s question, retrieve the most similar chunks, assemble them into a prompt, and send that to the model.
Chunking is where most projects go wrong
Chunking is the act of splitting documents into pieces small enough to retrieve and embed. It is also the step people give the least thought to and then wonder why their answers are bad. If your chunks are too large, each embedding becomes a blurry average of several topics and similarity search gets imprecise. If your chunks are too small, you retrieve fragments that lack the context needed to answer anything.
What works for me is chunking along the natural structure of the document. Split on headings and paragraphs rather than blindly every 500 characters, because a chunk that respects a section boundary carries a coherent idea. I also overlap chunks slightly so a sentence near a boundary is not orphaned from its context. A few hundred tokens per chunk with a small overlap is a reasonable starting point, but the right answer depends on your content, and you should look at actual chunks to sanity check them.
Ingestion in code
Here is the shape of an ingestion step using pgvector. I am keeping it deliberately small so the structure is visible.
import psycopg2
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-mpnet-base-v2")
conn = psycopg2.connect("dbname=app")
def ingest(doc_id, chunks):
vectors = model.encode(chunks, normalize_embeddings=True)
with conn.cursor() as cur:
for text, vec in zip(chunks, vectors):
cur.execute(
"INSERT INTO chunks (doc_id, body, embedding) "
"VALUES (%s, %s, %s)",
(doc_id, text, vec.tolist()),
)
conn.commit()
Nothing here is exotic. The interesting work happened before this function ran, in how the chunks were produced, and it happens after, in how you retrieve.
Retrieval and the prompt
At query time you embed the question with the same model you used for ingestion. Using a different model is a subtle and painful bug, because the two vector spaces do not line up and your similarity scores become meaningless. Then you retrieve the top candidates and build the prompt.
def answer(question, k=5):
qvec = model.encode([question], normalize_embeddings=True)[0]
with conn.cursor() as cur:
cur.execute(
"SELECT body FROM chunks "
"ORDER BY embedding <=> %s::vector LIMIT %s",
(qvec.tolist(), k),
)
context = "\n\n".join(row[0] for row in cur.fetchall())
prompt = (
"Answer using only the context below. "
"If the answer is not present, say you do not know.\n\n"
"Context:\n" + context + "\n\nQuestion: " + question
)
return call_model(prompt)
That instruction to say “I do not know” when the answer is not in the context is not optional. Without it the model will happily fill gaps with plausible fabrication, and a confident wrong answer is worse than no answer.
Retrieval quality decides everything
The model can only be as good as what you feed it. If retrieval surfaces the wrong chunks, no amount of prompt cleverness saves the answer. This is why I spend most of my RAG effort on retrieval rather than on prompt wording. A few techniques that earn their keep:
- Hybrid search. Combine vector similarity with keyword search. Semantic search misses exact identifiers, error codes, and product names, and keyword search catches them. Merging the two rankings beats either alone in almost every system I have measured.
- Reranking. Retrieve a generous set of candidates, then run a cross encoder reranker to reorder them by relevance before you build the prompt. The reranker is slower per item but far more accurate than raw vector distance, and applying it to a small candidate set is cheap.
- Metadata filtering. Restrict retrieval to the documents the user is allowed to see and that are recent enough to matter. This is both a relevance and a security concern.
The unglamorous problems
Real RAG systems live or die on the parts nobody demos. Keeping the index in sync when documents change, so you are not retrieving deleted or stale content. Handling access control so a user never retrieves a chunk from a document they cannot see, which is a genuine data leak if you get it wrong. Evaluating quality with a real test set of questions and expected answers rather than vibes, because without measurement you cannot tell whether a change helped. And managing the prompt size so you do not blow past the context window or pay for tokens you did not need.
The security dimension deserves more attention than it usually gets, especially once these systems start taking actions on a user’s behalf. I dug into that in my piece on agentic AI and cybersecurity, and the access control failures there map directly onto RAG retrieval.
Start simple, then measure
My advice is to build the simplest version first. Naive chunking, pgvector, top five retrieval, a clear prompt. Get it answering questions end to end. Then build an evaluation set and improve one thing at a time, measuring each change. Add hybrid search and check the numbers. Add reranking and check again. Tune chunking and check once more. The teams that succeed with RAG are not the ones with the fanciest stack. They are the ones who measure retrieval quality and grind on it. For the broader engineering context around shipping these systems, see my notes on practical AI engineering, and when you are ready to pick a store, choosing a vector database covers the options.