The RAG Pipeline

Retrieval-augmented generation gives a language model a temporary, query-specific memory. Before the model answers, another system searches an external corpus, selects evidence, and places that evidence in the prompt. The generator still predicts tokens; now those predictions can be conditioned on current, private, and attributable source material.

The useful work sits in the pipeline around the model. Documents must be parsed into retrieval units, embedded and indexed, found by several ranking signals, reranked with a more expensive model, fitted into a context budget, and traced through evaluation. A weak link anywhere produces a fluent answer backed by the wrong material.

This drawing treats RAG as production plumbing. Every stage has an input contract, an output contract, a cost, and a failure mode. The goal is a system you can inspect when the answer is wrong, not a single call that happens to look convincing.

A model's parameters compress patterns from training. They do not provide a dependable database of every training fact, and they cannot contain events or documents created after training. Private runbooks, current inventory, customer records, and today's policy need another path into the answer.

Longer context windows help when the relevant material is already known and fits. They do not decide which ten pages matter among ten million, keep an index current, or attach access controls to each passage. Retrieval supplies that selection layer.

Fine-tuning changes behavior: style, format, task habits, and sometimes domain performance. It is a poor database update mechanism because facts become diffuse in weights, provenance is hard to recover, and deletion is awkward. Retrieval keeps source material outside the model where it can be versioned, filtered, cited, and removed.

The lineage runs through open-domain question answering. DrQA, published by Chen and collaborators in 2017^[1], searched Wikipedia with lexical features and passed selected passages to a neural reader. The architecture already contained the enduring split: a fast document retriever narrowed the corpus, then a more expensive model read the shortlist for an answer. The reader could only succeed if retrieval put the evidence in reach.

Dense Passage Retrieval replaced that first-stage lexical representation with learned query and passage encoders in 2020^[2]. REALM trained a language model to retrieve from an external corpus as part of pretraining^[3]. Lewis and collaborators' 2020 RAG paper joined a neural retriever with a sequence generator and gave the pattern its durable name^[4]. These systems differed in training and generation details, but they shared one practical premise: knowledge can remain in an inspectable corpus and enter computation when a question arrives.

The economics follow the same split. Retrieval adds storage, embedding, and search costs on every corpus and query. Fine-tuning adds training and model-management costs. Neither is free; they buy different capabilities. Use retrieval when the answer depends on selecting external facts at request time.

Retrieval also changes the failure contract. A parameter-only answer may be difficult to trace beyond the prompt and model version. A retrieval-backed answer can preserve document ID, revision, passage offsets, and the exact evidence delivered to the model. That record does not guarantee correctness, but it creates something an operator can inspect and a user can verify.

RAG has an offline path and an online path. Offline, connectors read sources, parsers recover structure, a chunker creates retrieval units, an embedding model produces vectors, and an index stores vectors with text and metadata. Online, the user's query is transformed, searched, filtered, reranked, packed into a prompt, and sent to the generator.

The two paths run on different clocks. Ingestion may take seconds or hours after a source changes; the request path usually has a latency budget measured in fractions of a second to a few seconds. A document is searchable only after every offline stage succeeds. Monitoring "connector completed" is insufficient when parsing dropped a table, embedding failed for one batch, or the new index has not become queryable.

Every arrow needs identifiers and versions. A chunk should carry source ID, document version, location, access policy, parser version, chunker version, embedding model, and index build. A request trace should preserve the original query, transformed queries, candidate scores, reranker order, final context, model configuration, answer, and citations.

Freshness is an ingestion property. Updates must create or replace chunks; deletes must remove them; permission changes must propagate before search can expose stale access. Treat the index as a derived view of source systems, with reconciliation jobs and observable lag.

Idempotent ingestion makes recovery possible. Reprocessing the same source version should produce the same chunk identities or safely replace the earlier set. Tombstones or source-version manifests help remove chunks that disappeared after an edit. For sensitive corpora, permission lag deserves its own service objective because a stale allow rule can become a data exposure, while a stale deny rule becomes a support ticket.

The retriever returns chunks, so chunk boundaries become a hard limit on evidence quality. Fixed-size chunking is cheap and predictable but can split a definition from its explanation. Structure-aware chunking follows headings, paragraphs, lists, or code symbols. Semantic chunking detects topic shifts, adding model cost and another behavior to test.

Overlap copies text from the end of one chunk into the next. It protects facts near boundaries, increases embedding and storage cost, and can flood the top ranks with near-duplicates. Parent-child retrieval offers another pattern: index small child chunks for precise matching, then return a larger parent section for context.

Tables, code, and conversations expose the weakness of one universal chunk rule. A table row may be meaningless without its column headers. A function needs its signature and perhaps its enclosing class. A support reply can depend on the customer message above it. Structure-aware parsing should carry those relationships into text or metadata before the chunker starts counting tokens.

Character counts in the teaching panel make boundaries easy to see; production chunkers commonly count model tokens. Start from document structure, set a range rather than a sacred size, and evaluate with real questions. The RAG visualized walkthrough shows how one query moves through these handoffs.

Chunk size creates a precision-context trade. Small chunks give the retriever a focused target and reduce irrelevant prompt text, but may omit the qualifier that makes an answer safe. Large chunks preserve local narrative and consume more budget while diluting the embedding. Parent-child retrieval separates the two decisions: match against a focused child, then expand to a parent only after the match earns space.

Dense retrieval embeds query and chunks, then ranks by vector similarity. It handles paraphrases, conceptual matches, and multilingual similarity when the embedding model supports them. It can blur exact identifiers and rare terms because those strings occupy little of the pooled semantic representation.

Keyword retrieval, commonly BM25, rewards terms that appear in a passage and discounts terms common across the corpus. It excels at error codes, function names, legal clauses, product SKUs, and quoted language. It misses paraphrases when query and document vocabulary diverge.

BM25 belongs to the Okapi retrieval tradition and remains strong because its assumptions match many real queries. Term frequency helps when a query term appears repeatedly, document-length normalization prevents long documents from winning by accumulation alone, and inverse document frequency gives rare terms more influence. Robertson and Zaragoza's 2009 survey^[5] describes the probabilistic lineage and the family of weighting choices behind the familiar formula.

Dense Passage Retrieval demonstrated the complementary case: a learned bi-encoder could retrieve relevant Wikipedia passages when questions and answers used different surface language. Dense retrieval still has a vocabulary and training distribution, so it does not erase lexical failure. Hybrid search keeps both routes open.

Hybrid retrieval runs both and combines normalized scores or fuses ranks. The blend coefficient α below is an intuition aid; production systems often use reciprocal rank fusion^[6] because dense and BM25 score scales are unrelated.

Filters belong in this stage: tenant, permissions, language, product, time range, or document status. Apply security predicates inside retrieval whenever the store supports it. Post-filtering a small top-k can leave too few legal results and can expose forbidden text to application memory or logs.

Query rewriting can help either branch and can also destroy signal. Expanding "SSO loop" into a fuller diagnostic query may surface prose that uses repeated authentication redirect. Replacing the user's exact error code with a paraphrase can remove the best BM25 term. Preserve the original query, search rewritten variants alongside it, and evaluate the fusion rather than assuming a more articulate query is better.

Embedding search uses a bi-encoder: query and passage are encoded separately, so passage vectors can be precomputed. That makes search fast enough for millions of candidates, but query and passage cannot inspect each other's tokens during scoring^[7].

A cross-encoder reads the query and one candidate together. Full token-level attention produces a more precise relevance judgment at much higher cost. The common cascade retrieves tens or hundreds of candidates cheaply, reranks that short list, and sends only the best few passages to the prompt.

The cost gap comes from reuse. A bi-encoder computes each passage vector during ingestion and performs one cheap comparison at request time. A cross-encoder must run a fresh joint forward pass for every query-passage pair. Reranking 100 candidates means roughly 100 pair evaluations, though batching improves hardware use. Applying that model to a million passages per query would throw away the reason an index exists.

Rerankers can also enforce policies that vector similarity cannot express cleanly: source authority, recency, document type, diversity, or whether a candidate contains enough context to answer. Keep learned relevance separate from explicit business rules so each can be inspected.

Diversity matters when the first page contains five overlapping chunks from one document. Sending all five spends budget on repeated sentences and may crowd out a second source that confirms or challenges the first. Deduplication by source offsets, maximal marginal relevance, or a simple per-document cap can improve the evidence packet even when raw relevance scores fall slightly.

The assembly stage turns ranked chunks into a bounded evidence packet. Reserve tokens for system instructions, conversation history, tool schemas, the user's question, and the model's answer. What remains is the retrieval budget. Adding passages until the window is full can reduce quality by introducing conflicts and distracting material.

Ordering matters. Models can underuse evidence buried in the middle of a long context — the lost-in-the-middle effect. Put the strongest material where the chosen model follows it reliably, group related chunks, remove duplicates, and preserve source boundaries. Tell the model how to cite, how to handle conflicts, and to abstain when evidence is insufficient.

Liu and collaborators measured this behavior in 2023^[8] by moving relevant information through long inputs. Performance was often strongest when evidence appeared near the beginning or end and weaker when it sat in the middle, even for models whose context window covered the full input. A context window tells you how many tokens the model can accept. It does not promise equal use of every position.

Compression is another model call with another failure surface. Extractive compression keeps selected source sentences and preserves provenance more cleanly. Abstractive summaries save more tokens but can erase qualifications or invent connective claims. Evaluate the compressed evidence, not only the final prose.

Citations need a mechanical path back to chunks. Give each context block a stable source label and instruct the model to attach those labels to claims. After generation, verify that every cited label was present in the supplied context and resolve it to the original document location. This catches fabricated identifiers and broken links, though checking whether the cited passage truly supports the sentence requires a separate faithfulness judgment.

Multi-hop questions complicate packing because no single passage contains the answer. One chunk may identify a product's owning team, while another gives that team's escalation policy. A single retrieval pass can miss the bridge between them. Query decomposition or iterative retrieval can find each step, but every added round increases latency and creates another place for an early mistake to steer later searches.

When an answer is wrong, first ask whether the required evidence existed in the corpus. Then ask whether ingestion parsed it, retrieval found it, reranking kept it, assembly included it, and generation used it. A single end-to-end score hides which component needs work.

That trace suggests a practical debugging order. Start with the source of truth and expected passage, then inspect each artifact produced downstream. If exact search cannot find the expected chunk, fix parsing or chunking. If exact search finds it but ANN does not, tune the index. If retrieval succeeds and the passage vanishes after reranking or assembly, the generator is innocent of that miss.

Retrieval metrics need labeled relevance. Hit rate@k asks whether at least one relevant item appeared. Recall@k asks how much of the relevant set appeared. MRR rewards putting the first relevant result high. nDCG handles graded relevance and position. Measure latency and filtered-result counts beside quality.

Generation metrics separate faithfulness — are claims supported by supplied context? — from answer relevance — does the response address the question? Context relevance measures whether retrieved material is useful rather than merely related. This RAG triad catches the fluent answer that ignores evidence and the faithful answer built from irrelevant evidence.

Build a small human-reviewed evaluation set before tuning. Include answerable questions, unanswerable questions, exact identifiers, paraphrases, permission boundaries, stale documents, conflicting sources, tables, and multi-hop cases. Model-based judges can scale review, but calibrate them against humans and retain examples behind every aggregate score.

Evaluation sets age with the corpus. A policy question can change its correct passage after a revision, and a once-unanswerable question can become answerable when a document arrives. Store expected source versions and relevance labels, review disagreements, and keep a stable regression slice beside a rotating sample of current traffic. Production feedback is useful evidence, but clicks and thumbs-up reflect presentation and user expectations as well as retrieval quality.

Abstention deserves direct tests. Include questions whose answer is absent, whose sources conflict, and whose only matching document is outside the user's permissions. Score whether the system declines cleanly and whether its explanation avoids revealing restricted details. An answer-quality average can hide a system that performs well on easy questions and improvises dangerously when evidence is missing.

Use plain long context when the source set is small, already selected, and fits comfortably. A contract reviewer analyzing one uploaded agreement may need careful parsing and prompting more than a persistent retrieval index. Search adds failure modes when there is nothing meaningful to search.

Use tools when the task needs actions or exact live state. A database query, calculator, filesystem read, or API call can return authoritative structured results. An agent may still use retrieval for documentation, but vector search should not impersonate a transactional system.

Use fine-tuning when the target is behavior: stable tone, format, classification policy, or repeated task procedure. Retrieval can provide examples and facts, yet stuffing many demonstrations into every prompt may cost more and behave less consistently than a tuned model.

Use a direct structured lookup when the question has one authoritative row. Account balance, shipment status, and permission checks belong behind APIs or database queries with typed results. Retrieval can help locate the policy explaining a balance, while the live amount should come from the transactional source. The two results can share one answer without pretending they offer the same guarantee.

These choices compose. A support agent can retrieve policy, call an order API, and answer in a fine-tuned format. The architecture should follow the information source and required guarantee for each step.

The working pipeline ends where it began: with evidence. Keep sources current, make retrieval measurable, preserve provenance through prompt assembly, require the generator to stay inside the supplied record, and expose every handoff in the trace. Then a wrong answer becomes an engineering problem with a location.