Retrieval-Augmented Generation (RAG 101)

A language model can generate fluent text from patterns encoded in its parameters. It cannot automatically inspect your current refund policy, private repository, incident log, or product catalogue. Even when relevant facts were present during training, the model may not reproduce them accurately or show where they came from.

Retrieval-Augmented Generation adds an explicit knowledge path. Before generation, the application searches an external corpus, selects useful evidence, and supplies that evidence with the question. The original RAG paper described a model combining parametric memory with a dense, non-parametric index. In production engineering, “RAG” now commonly describes the wider retrieve-then-generate system around an LLM.

This guide uses a company refund-policy question as a running example. The architecture is the same for support assistants, code search, compliance workflows, research tools, and internal knowledge systems.

A clean RAG design separates work that prepares knowledge from work that answers a request. The indexing pipeline can run on a schedule or react to source changes. The answer pipeline must execute within a user-facing latency budget.

• Offline indexing: ingest sources, preserve structure, attach metadata and access controls, create searchable representations, and update or delete them when the source changes.
• Online answering: understand the question, retrieve candidates the user may access, rerank them, assemble a bounded context, generate an answer, and return inspectable sources.

This boundary is operationally useful. You can re-index a source without changing the answer API, replace a reranker without re-parsing documents, and evaluate retrieval separately from generation.

The job of indexing is not to turn every file into equal-sized strings. It is to preserve enough meaning, provenance, and permission data that the right passage can be found later.

1. Parse with structure. Keep headings, lists, tables, page references, code boundaries, and source identifiers. A format-aware parser is usually more valuable than another prompt tweak.
2. Chunk around meaning. Use section boundaries, semantic units, or task-specific rules. There is no universal “few hundred tokens” setting; chunk size is an evaluation parameter.
3. Attach metadata and ACLs. Store source, revision, owner, timestamp, document type, and access-control attributes with every retrievable unit.
4. Create complementary indexes. Dense embeddings help with semantic similarity. Lexical search protects exact names, identifiers, error codes, and domain language. Many production systems combine both.
5. Treat updates and deletion as first-class. A stale chunk with a strong similarity score is still the wrong answer.

for document in source.changed_documents():
    parsed = parser.parse(document)

    for chunk in chunker.split(parsed):
        index.upsert(
            id=stable_id(document, chunk),
            text=chunk.text,
            vector=embed(chunk.text),
            metadata={
                "source": document.url,
                "revision": document.revision,
                "acl": document.allowed_principals,
            },
        )

for document_id in source.deleted_documents():
    index.delete(where={"document_id": document_id})

When a user asks “What is our refund policy?”, the application turns that request into one or more search queries. A first-stage retriever searches broadly. A second-stage reranker spends more compute on a smaller candidate set. Context assembly then selects the best passages that fit the model’s input budget.

The funnel matters because nearest-neighbour search is only a candidate generator. A high cosine-similarity score does not prove that a passage is current, authoritative, complete, or appropriate for the task.

def answer(question: str, principal: Principal) -> Answer:
    queries = query_planner.expand(question)

    candidates = retriever.search(
        queries=queries,
        filters={"acl": principal.effective_groups},
        limit=40,
    )

    ranked = reranker.rank(question, candidates)
    context = context_builder.pack(ranked, token_budget=6_000)

    return generator.generate(
        question=question,
        evidence=context,
        require_citations=True,
    )

RAG and Transformer attention operate at different boundaries. Retrieval runs outside the generator and decides which external evidence becomes input. Self-attention runs inside the Transformer and computes how the tokens already in that input influence one another during representation and generation.

The distinction is practical. If the policy document never reaches the prompt, changing attention cannot recover it. If retrieval returns the correct passage but the response ignores or misinterprets it, the failure is in context construction, prompting, or generation rather than search.

Advanced RAG patterns are responses to specific failure modes. They are not maturity levels that every system must climb.

• Naive RAG uses one query, one retrieval pass, and one generation step. It is a strong baseline when the corpus and questions are narrow.
• Hybrid and fusion retrieval combine semantic and lexical candidates, sometimes using multiple query formulations and reciprocal-rank fusion before reranking.
• Corrective RAG evaluates retrieval quality and chooses a corrective action when evidence is weak. The original CRAG paper describes confidence-based retrieval actions, web-search augmentation, and knowledge refinement.
• Agentic RAG lets a planner decide when and where to retrieve, inspect results, and issue follow-up searches. It adds flexibility along with more latency, cost, and failure states.
• Graph RAG represents entities and relationships explicitly for questions that require connections or corpus-level summaries. Microsoft’s GraphRAG documentation describes an indexing pipeline that extracts entities, relationships, claims, and community summaries.

The useful decision is not which library wins. It is where your team wants to own complexity. These categories are representative examples as of July 2026, not endorsements.

Whichever layer you choose, keep your application contracts explicit: a parser produces structured chunks, a retriever returns evidence with provenance, a reranker returns scores, and a generator receives a bounded evidence set. Those seams make replacements and evaluation possible.

A demo is complete when it answers one question. A production RAG system must keep answering across changing documents, permissions, user language, and provider failures.

• Retrieval quality: measure recall at k, ranking quality, empty-result rate, and which source types are consistently missed.
• Answer quality: measure groundedness, citation correctness, completeness, task success, and abstention when evidence is insufficient.
• Freshness and deletion: track ingestion lag, failed syncs, tombstones, and the revision attached to every cited passage.
• Security: propagate identity and ACLs, isolate tenants, defend against prompt injection in retrieved text, and avoid logging sensitive context by default.
• Reliability: define timeouts and fallbacks for parsers, embedding services, indexes, rerankers, and generators. A partial answer with clear limits is better than invented certainty.
• Cost and latency: observe each stage independently. Candidate count, reranking depth, context size, and model choice are product-quality controls as well as cost controls.