khalil Drissi Cms

Tag: RAG

  • Practical AI engineering: shipping LLM features that hold up

    Practical AI engineering: shipping LLM features that hold up

    There is a wide gap between a demo that works in front of an audience and a feature that survives real users for a month. I have shipped a few LLM features now, and almost everything I learned the hard way lives in that gap.

    The demo is the easy 80 percent

    Wiring up a model and getting a good answer takes an afternoon. The remaining work is everything that happens when the input is weird, the model is confidently wrong, or the user asks something you never tested. That part takes the other three weeks, and it is the part that decides whether anyone keeps using the thing.

    So plan for it. Budget more time for evaluation and guardrails than for the happy path, because the happy path mostly builds itself.

    RAG: retrieval is the hard part, not generation

    Most useful LLM features need your data, not just the model’s training. Retrieval-augmented generation is the standard answer: find the relevant chunks, put them in the prompt, let the model answer from them. Simple to describe, fiddly to get right.

    The quality of a RAG system is almost entirely the quality of its retrieval. If you fetch the wrong chunks, no amount of prompt cleverness saves you. Spend your time on chunking strategy, on whether you actually need embeddings or whether plain keyword search wins for your data, and on measuring whether the retrieved context contains the answer before you ever look at the generation step.

    One concrete tip: log the retrieved chunks for every query in development. Half my RAG bugs were obvious the moment I saw what the retriever actually pulled.

    You cannot improve what you do not measure

    “It seems better” is not a metric. Before you tune anything, build a small evaluation set: thirty to fifty real inputs with known good outputs. Run it on every change. It feels like overkill until the day a prompt tweak that “obviously improved things” quietly broke a third of your cases.

    Evals do not need to be fancy. A spreadsheet of inputs, expected behavior, and a pass or fail you check by eye beats no evals at all. Automate it later once you know what you are measuring.

    Treat the model output as untrusted

    This is the lesson that connects to security. Model output is just text, and if you feed it into a database query, a shell command, or another system, it can do damage the same way user input can. If an agent reads untrusted content, that content can carry instructions, which is the prompt-injection problem I cover in agentic AI in cybersecurity.

    Validate structured output against a schema. Never pass raw model text into anything that executes. The same “input is hostile” mindset from my developer security checklist applies directly to what comes out of the model, not just what goes in.

    Cost and latency are product decisions

    The biggest model is rarely the right default. A smaller model that answers in 400 milliseconds often beats a larger one that takes four seconds, because users feel latency immediately and judge quality slowly. Cache aggressively. Route easy queries to cheap models and save the expensive one for the hard cases.

    Pick your model tier on purpose. I default to the most capable model while building, then drop down once I know which calls actually need the horsepower.

    Where this leaves you

    Shipping AI features is mostly normal engineering with a probabilistic component bolted on. The model is the fun part and the smallest part. Retrieval, evaluation, validation, and the plumbing around it are the job. If you are building the surrounding system from scratch, the patterns in modern full-stack architecture are where the model actually has to live.

  • Building RAG systems with vector databases

    Building RAG systems with vector databases

    Retrieval augmented generation sounds complicated and is actually simple in outline. You give a language model access to your documents by retrieving the relevant ones and pasting them into the prompt. The model answers using that context instead of relying only on what it memorized during training. The outline fits in a sentence. The reason RAG systems fail is never the outline. It is the details, and this post is about the details, because I have watched the same mistakes sink the same projects more than once.

    This assumes you understand embeddings and similarity search already. If not, read vector databases explained first, because the entire retrieval step depends on those ideas.

    The pipeline at a glance

    A RAG system has two phases. There is an offline ingestion phase where you process your documents and store them, and an online query phase where you answer a user’s question. Ingestion looks like this: take your documents, split them into chunks, generate an embedding for each chunk, and store the vectors with their text and metadata in a vector database. Querying looks like this: embed the user’s question, retrieve the most similar chunks, assemble them into a prompt, and send that to the model.

    Chunking is where most projects go wrong

    Chunking is the act of splitting documents into pieces small enough to retrieve and embed. It is also the step people give the least thought to and then wonder why their answers are bad. If your chunks are too large, each embedding becomes a blurry average of several topics and similarity search gets imprecise. If your chunks are too small, you retrieve fragments that lack the context needed to answer anything.

    What works for me is chunking along the natural structure of the document. Split on headings and paragraphs rather than blindly every 500 characters, because a chunk that respects a section boundary carries a coherent idea. I also overlap chunks slightly so a sentence near a boundary is not orphaned from its context. A few hundred tokens per chunk with a small overlap is a reasonable starting point, but the right answer depends on your content, and you should look at actual chunks to sanity check them.

    Ingestion in code

    Here is the shape of an ingestion step using pgvector. I am keeping it deliberately small so the structure is visible.

    import psycopg2
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-mpnet-base-v2")
conn = psycopg2.connect("dbname=app")

def ingest(doc_id, chunks):
    vectors = model.encode(chunks, normalize_embeddings=True)
    with conn.cursor() as cur:
        for text, vec in zip(chunks, vectors):
            cur.execute(
                "INSERT INTO chunks (doc_id, body, embedding) "
                "VALUES (%s, %s, %s)",
                (doc_id, text, vec.tolist()),
            )
    conn.commit()

    Nothing here is exotic. The interesting work happened before this function ran, in how the chunks were produced, and it happens after, in how you retrieve.

    Retrieval and the prompt

    At query time you embed the question with the same model you used for ingestion. Using a different model is a subtle and painful bug, because the two vector spaces do not line up and your similarity scores become meaningless. Then you retrieve the top candidates and build the prompt.

    def answer(question, k=5):
    qvec = model.encode([question], normalize_embeddings=True)[0]
    with conn.cursor() as cur:
        cur.execute(
            "SELECT body FROM chunks "
            "ORDER BY embedding <=> %s::vector LIMIT %s",
            (qvec.tolist(), k),
        )
        context = "\n\n".join(row[0] for row in cur.fetchall())
    prompt = (
        "Answer using only the context below. "
        "If the answer is not present, say you do not know.\n\n"
        "Context:\n" + context + "\n\nQuestion: " + question
    )
    return call_model(prompt)

    That instruction to say “I do not know” when the answer is not in the context is not optional. Without it the model will happily fill gaps with plausible fabrication, and a confident wrong answer is worse than no answer.

    Retrieval quality decides everything

    The model can only be as good as what you feed it. If retrieval surfaces the wrong chunks, no amount of prompt cleverness saves the answer. This is why I spend most of my RAG effort on retrieval rather than on prompt wording. A few techniques that earn their keep:

    • Hybrid search. Combine vector similarity with keyword search. Semantic search misses exact identifiers, error codes, and product names, and keyword search catches them. Merging the two rankings beats either alone in almost every system I have measured.
    • Reranking. Retrieve a generous set of candidates, then run a cross encoder reranker to reorder them by relevance before you build the prompt. The reranker is slower per item but far more accurate than raw vector distance, and applying it to a small candidate set is cheap.
    • Metadata filtering. Restrict retrieval to the documents the user is allowed to see and that are recent enough to matter. This is both a relevance and a security concern.

    The unglamorous problems

    Real RAG systems live or die on the parts nobody demos. Keeping the index in sync when documents change, so you are not retrieving deleted or stale content. Handling access control so a user never retrieves a chunk from a document they cannot see, which is a genuine data leak if you get it wrong. Evaluating quality with a real test set of questions and expected answers rather than vibes, because without measurement you cannot tell whether a change helped. And managing the prompt size so you do not blow past the context window or pay for tokens you did not need.

    The security dimension deserves more attention than it usually gets, especially once these systems start taking actions on a user’s behalf. I dug into that in my piece on agentic AI and cybersecurity, and the access control failures there map directly onto RAG retrieval.

    Start simple, then measure

    My advice is to build the simplest version first. Naive chunking, pgvector, top five retrieval, a clear prompt. Get it answering questions end to end. Then build an evaluation set and improve one thing at a time, measuring each change. Add hybrid search and check the numbers. Add reranking and check again. Tune chunking and check once more. The teams that succeed with RAG are not the ones with the fanciest stack. They are the ones who measure retrieval quality and grind on it. For the broader engineering context around shipping these systems, see my notes on practical AI engineering, and when you are ready to pick a store, choosing a vector database covers the options.