Production RAG beyond vector search

A retrieval-augmented generation demo is pleasantly easy to build. Split a few documents, create embeddings, retrieve the nearest chunks, place them in a prompt, and ask a model to answer. The result often looks convincing enough to trigger the dangerous question: “How quickly can we put this into production?”

The difficult work starts after the demo. Production RAG is not a vector-search feature. It is a changing information system with probabilistic components, asynchronous data pipelines, access-control obligations, and several independent ways to produce a confident but unhelpful answer.

This article develops a practical mental model for that system. The focus is not a specific framework or database. It is the set of boundaries and measurements that keep the design understandable as data, traffic, models, and user expectations change.

Start with the answer contract

Before choosing an embedding model, write down what an acceptable answer means. This sounds obvious, but many teams begin by optimizing retrieval without agreeing on the product behavior retrieval is supposed to support.

An answer contract should clarify:

• which sources the system may use;
• how fresh those sources must be;
• whether every material claim requires a citation;
• what the system does when evidence is missing or contradictory;
• which users may see which pieces of evidence;
• and how much latency and cost the interaction can tolerate.

For an internal policy assistant, “useful” may mean citing the current approved policy and refusing to infer beyond it. For customer support, it may mean combining account context with public documentation while never exposing another tenant’s information. Those are different systems even if their diagrams contain the same boxes.

Treat ingestion as a product pipeline

Online retrieval can only be as reliable as the indexed representation it receives. Ingestion therefore deserves its own ownership, service-level objectives, and observability.

A useful ingestion pipeline normally separates these stages:

1. Acquire the source and preserve its identity, version, permissions, and timestamps.
2. Parse the source into a stable intermediate representation rather than coupling every downstream step to PDF, HTML, or document-specific behavior.
3. Enrich the representation with headings, entities, language, document type, and other metadata useful for filtering or ranking.
4. Segment content using its structure and retrieval purpose rather than a universal character count.
5. Embed and index the resulting units with an explicit model and schema version.
6. Validate counts, permissions, empty content, duplication, and index visibility before declaring the version available.

Each stage should be replayable. When the chunking strategy or embedding model changes, the team should be able to build a new index version alongside the current one, evaluate it, and switch traffic deliberately. Rewriting the active index in place removes the ability to compare or roll back.

Freshness is observable state

“The documents are indexed” is not a sufficient operational signal. Track source version, discovered time, processing time, published index version, and the lag between them. A green ingestion job can still serve stale information if discovery silently stopped upstream.

At minimum, expose:

source_version
discovered_at
processed_at
index_version
index_visible_at
permission_version

This makes freshness measurable per source and allows the product to communicate when information is still being processed.

Retrieval is a staged decision

Nearest-neighbour search is one candidate-generation technique, not the complete retrieval strategy. Production retrieval is usually a staged pipeline:

Query understanding
  → permission and metadata filters
  → candidate generation
  → reranking
  → evidence selection
  → context assembly

Query understanding may normalize terminology, identify an entity or date range, decide that the question needs multiple searches, or determine that retrieval should not run at all. Metadata filters remove ineligible material before semantic similarity is considered. Candidate generation can combine lexical and vector results. A reranker then spends more computation on a small candidate set, while evidence selection manages redundancy and context budget.

Preserve permission boundaries early

Do not retrieve unauthorized chunks and hope to remove them before generation. Apply tenant and access filters during candidate generation, and carry source authorization metadata through every derived representation.

If a document’s permissions change, the system needs a defined propagation path. Depending on the risk, that may require immediate filtering against an authoritative permission service rather than waiting for asynchronous re-indexing.

Separate retrieval quality from answer quality

When an answer is wrong, teams often change the prompt because the model is the most visible component. The failure may have happened much earlier.

Evaluate the system at distinct boundaries:

• Candidate recall: did retrieval surface the evidence needed to answer?
• Ranking quality: did useful evidence appear near the top?
• Context quality: was the assembled context relevant, sufficiently diverse, and within budget?
• Groundedness: are the answer’s material claims supported by the provided evidence?
• Answer usefulness: is the response correct, complete, clear, and appropriate for the user’s task?

These measurements lead to different fixes. Poor recall may require query expansion or a better index. Poor ranking may require metadata, hybrid retrieval, or a reranker. Good context with an unsupported answer points toward generation policy or model behavior.

Build an evaluation set from real work

Synthetic questions are useful for bootstrapping, but they should not be the only evaluation source. Capture representative questions, difficult terminology, ambiguous requests, freshness-sensitive cases, permission boundaries, and legitimate no-answer scenarios.

Every evaluation case should contain more than a preferred answer. Record the expected evidence, important exclusions, user or tenant context, and the reason the case matters. That allows retrieval changes to be evaluated without depending entirely on a model judge.

Put policy in an orchestrator

The online path benefits from an explicit orchestration boundary. Its responsibility is not to become an all-knowing framework. It owns the product policy connecting otherwise replaceable capabilities.

The orchestrator decides:

• whether the request is eligible for retrieval;
• which retrieval strategy and corpus to use;
• how much latency and context budget remain;
• what happens when retrieval, reranking, or generation degrades;
• which evidence and evaluation metadata are retained;
• and whether the answer should be returned, revised, or declined.

Provider-specific concerns belong behind narrower boundaries. A model gateway can own provider routing, credentials, quotas, retries, and cost attribution. Retrieval services can evolve their indexes and ranking strategies. The orchestrator keeps the end-to-end decision visible.

Design degradation before failure

Dependencies rarely fail in a clean binary state. They become slow, return partial results, hit a quota, or produce a result that passes protocol validation but fails product expectations.

Consider explicit behavior for:

Retries need a budget and an owner. If the gateway, orchestrator, SDK, and queue all retry independently, a small provider incident becomes a traffic multiplier.

Observe a decision, not only a request

Traditional request metrics remain necessary: latency, traffic, errors, and saturation. RAG also needs enough structured trace data to reconstruct the decisions behind an answer without indiscriminately logging sensitive prompts and documents.

A trace might connect:

request_id
tenant_id
policy_version
query_strategy
index_version
retrieved_chunk_ids
reranker_version
model_route
prompt_template_version
token_usage
evaluation_results
degradation_state

Retention and redaction must match the data classification. Identifiers and hashes can often provide reproducibility without copying complete source text into every telemetry system.

Dashboards should combine system and quality signals. A fast response with declining evidence coverage is not healthy. Neither is a highly grounded response whose cost or latency makes the product unusable.

Evolve with versioned boundaries

RAG systems change continuously: source schemas, parsers, chunking rules, embedding models, indexes, rerankers, prompt templates, model providers, and evaluation policies. Version those boundaries explicitly.

For an important change:

1. build the new representation alongside the old one;
2. run the maintained evaluation set;
3. shadow a sample of production queries where policy permits;
4. compare quality, latency, cost, and failure behavior;
5. canary the new version;
6. retain a fast rollback path.

This is more deliberate than updating a library and watching aggregate errors. It also makes improvement attributable. Without versioned traces, teams cannot confidently say which change improved or damaged the product.

A practical production checklist

Before calling a RAG system production-ready, I would want clear answers to these questions:

• What is the answer contract, including refusal behavior?
• How is source identity, version, and permission preserved?
• Can ingestion stages be replayed independently?
• Can an index change be evaluated and rolled back?
• Are lexical and semantic candidates both considered where useful?
• Are permissions enforced before evidence reaches generation?
• Can retrieval and generation quality be measured separately?
• Does the evaluation set include real, difficult, and no-answer cases?
• Is degradation explicit and constrained by an end-to-end budget?
• Can an operator reconstruct the versions and evidence behind an answer?
• Are sensitive traces minimized, redacted, and retained intentionally?
• Can model and retrieval providers be changed without rewriting product policy?

The objective is not to create the most elaborate pipeline. It is to make the system’s uncertainty, boundaries, and evolution manageable. Start with the simplest design that satisfies the answer contract, then add complexity only when evaluation or operating evidence justifies it.

Vector search remains useful. It is simply not the architecture. The architecture is the set of decisions that turns changing information and probabilistic behavior into a product people can depend on.