Architecting Production-Grade Retrieval-Augmented Generation (RAG) at Scale

Retrieval-augmented generation looks simple in a demo: embed documents, search, stuff context into a prompt. At enterprise scale — millions of documents, permission boundaries, adversarial content — naive RAG fails quietly. Production RAG is a search engineering problem first.

Reference architecture

Sources ─► Parsing/OCR ─► Chunking ─► Embedding ─► Vector index
                                    └──────► BM25 index
Query ─► Rewrite ─► Hybrid search ─► Re-rank ─► ACL filter ─► LLM ─► Cited answer

Chunking is a modeling decision

Fixed 512-token windows break tables and split definitions from their context. Use structure-aware chunking that respects headings, lists and tables, and attach parent-section metadata so the generator sees enough surrounding context.

Hybrid retrieval and re-ranking

Dense embeddings capture semantics; BM25 captures exact identifiers like part numbers and policy codes. Fuse them with reciprocal rank fusion, then apply a cross-encoder re-ranker to the top 50–100 candidates.

def rrf(rankings, k=60):
    scores = defaultdict(float)
    for ranking in rankings:
        for rank, doc in enumerate(ranking):
            scores[doc] += 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Permission-aware retrieval

Filter by access control lists inside the vector query, not after generation. Post-filtering risks leaking restricted content into the prompt.

Evaluate retrieval and generation separately

• Retrieval: recall@k and MRR on a labeled query set.
• Generation: faithfulness to retrieved context and answer relevance.
• End-to-end: human-rated correctness on a stratified sample of real queries.

Most RAG quality problems are retrieval problems. Instrument both stages so you know which to fix.