The RAG Pipeline

Retrieval-augmented generation gives a language model a temporary memory, built fresh for each question. Before the model answers, a separate system searches an external corpus, picks out evidence, and drops that evidence into the prompt. The generator is still just predicting tokens. What's changed is that the predictions can now be conditioned on material that's current, private, and traceable back to a source.

Almost all of the engineering sits in the plumbing around the model. Documents have to be parsed into retrieval units, embedded, indexed, found by more than one ranking signal, reranked with something more expensive, squeezed into a context budget, and traced through evaluation. Break any one of those links and you get a fluent answer backed by the wrong material — which is the worst failure mode there is, because it looks fine.

This drawing treats RAG as production plumbing. Every stage has an input contract, an output contract, a cost, and a way of failing. What you want at the end is a system you can open up when the answer is wrong, rather than a single call that happened to sound convincing.

A model's parameters compress patterns out of its training data. They aren't a dependable database of every fact that went in, and they can't contain anything that happened afterward. Private runbooks, current inventory, customer records, this morning's policy change — all of that needs another route into the answer.

Longer context windows help when you already know which material is relevant and it fits. They won't tell you which ten pages matter out of ten million, keep an index current, or attach an access policy to each passage. Retrieval is the selection layer that does.

Fine-tuning changes behavior — style, format, task habits, sometimes domain performance. As a mechanism for updating facts it's poor: the facts diffuse into the weights, provenance is hard to recover afterward, and deleting one thing is awkward. Retrieval keeps the source material outside the model, where you can version it, filter it, cite it, and delete it.

The lineage runs through open-domain question answering. DrQA, published by Chen and collaborators in 2017^[1], searched Wikipedia with lexical features and handed the selected passages to a neural reader. The split that survived everything since is already there: a fast retriever narrows the corpus, then a more expensive model reads the shortlist. The reader could only succeed if retrieval put the evidence within reach.

Dense Passage Retrieval swapped that first-stage lexical representation for learned query and passage encoders in 2020^[2]. REALM trained a language model to retrieve from an external corpus as part of pretraining^[3]. Lewis and collaborators' 2020 paper joined a neural retriever to a sequence generator and gave the pattern the name it still has^[4]. The training and generation details differ across all of them. The shared premise is what stuck: knowledge can stay in a corpus you can inspect, and enter the computation only when a question shows up.

The economics split the same way. Retrieval bills you for storage, embedding, and search, on every corpus and every query. Fine-tuning bills you for training runs and for managing another model. Neither is free, and they buy different things. Retrieval is what you want when the answer depends on selecting external facts at request time.

Retrieval also changes the failure contract. When a parameter-only answer goes wrong, there's often nothing to trace beyond the prompt and the model version. A retrieval-backed answer can carry the document ID, the revision, the passage offsets, and the exact evidence that reached the model. That record doesn't make the answer correct. It does give an operator something to open and a user something to check.

RAG has an offline path and an online path. Offline, connectors read the sources, parsers recover structure, a chunker cuts retrieval units, an embedding model turns them into vectors, and an index stores those vectors alongside the text and metadata. Online, the user's query gets transformed, searched, filtered, reranked, packed into a prompt, and sent to the generator.

The two paths run on completely different clocks. Ingestion might lag a source change by seconds or by hours. The request path is usually working against a latency budget somewhere between a fraction of a second and a few seconds. And a document is only searchable once every single offline stage has succeeded, which is why "connector completed" is such a weak signal — it's still true when parsing dropped a table, when one embedding batch failed, or when the new index hasn't become queryable yet.

Every arrow on that diagram needs identifiers and versions attached to it. A chunk should carry its source ID, the document version it came from, where in the document it sits, its access policy, and the versions of the parser, chunker, embedding model, and index build that produced it. A request trace should keep the original query, whatever the rewriter turned it into, the candidate scores, the reranker's ordering, the final assembled context, the model configuration, the answer, and the citations.

Freshness is a property of ingestion, not of search. Updates have to create or replace chunks, deletes have to remove them, and permission changes have to propagate before search can expose something the user shouldn't see. The index is a derived view of the source systems, so give it reconciliation jobs and make its lag observable.

Idempotent ingestion is what makes recovery possible. Reprocess the same source version and you should get the same chunk identities back, or a clean replacement of the earlier set. Tombstones or source-version manifests handle the chunks that disappeared in an edit. On a sensitive corpus, permission lag deserves a service objective of its own — a stale allow rule turns into a data exposure, while a stale deny rule only turns into a support ticket.

The retriever returns chunks, which makes chunk boundaries a hard ceiling on evidence quality. Fixed-size chunking is cheap and predictable and will happily split a definition away from its explanation. Structure-aware chunking follows the headings, paragraphs, lists, or code symbols already in the document. Semantic chunking detects topic shifts, at the cost of another model call and another behavior you have to test.

Overlap copies text from the end of one chunk into the start of the next. It protects facts that land near a boundary, it costs more to embed and store, and it can flood your top ranks with near-duplicates of each other. Parent-child retrieval is the other way out: index small child chunks so matching stays precise, then return the larger parent section once a child matches.

Tables, code, and conversations are where one universal chunk rule falls apart. A table row means nothing without its column headers. A function needs its signature, and probably its enclosing class. A support reply can be unintelligible without the customer message above it. Structure-aware parsing has to carry those relationships into the text or the metadata before the chunker starts counting tokens, because the chunker itself has no idea they exist.

The teaching panel counts characters so the boundaries are easy to see; production chunkers usually count model tokens instead. Start from the document's own structure, pick a range rather than a sacred number, and evaluate against questions people actually ask. The RAG visualized walkthrough follows one query through these handoffs.

Chunk size buys precision at the cost of context. Small chunks give the retriever a focused target and keep irrelevant text out of the prompt, and they can drop the qualifier that made the answer safe. Large chunks preserve the local narrative, eat more budget, and dilute the embedding. Parent-child retrieval is attractive because it separates the two decisions: match against a focused child, and only expand to the parent once the match has earned the space.

Dense retrieval embeds the query and the chunks, then ranks by vector similarity. It handles paraphrases, conceptual matches, and cross-language similarity, assuming the embedding model was trained for it. Where it struggles is exact identifiers and rare terms, because those strings occupy very little of a pooled semantic representation.

Keyword retrieval, usually BM25, rewards terms that show up in a passage and discounts terms that show up everywhere in the corpus. Error codes, function names, legal clauses, product SKUs, quoted language — that's where it shines. It misses paraphrases whenever the query and the document reach for different words.

BM25 comes out of the Okapi retrieval tradition, and it stays strong because its assumptions still match a lot of real queries. Term frequency helps when a query term repeats in a passage. Document-length normalization stops long documents from winning by sheer accumulation. Inverse document frequency gives the rare terms more pull. Robertson and Zaragoza's 2009 survey^[5] traces the probabilistic lineage and the family of weighting choices sitting behind the formula everyone copies.

Dense Passage Retrieval demonstrated the complementary case: a learned bi-encoder could pull up the right Wikipedia passage when the question and the answer used entirely different surface language. Dense retrieval has a vocabulary and a training distribution of its own, though, so it doesn't abolish lexical failure — it relocates it. Hybrid search is how you keep both routes open.

Hybrid retrieval runs both and either combines normalized scores or fuses the ranks. The blend coefficient α below is an intuition aid more than a recipe; production systems tend to use reciprocal rank fusion^[6] instead, because dense and BM25 scores live on unrelated scales.

Filters belong in this stage — tenant, permissions, language, product, time range, document status. Push the security predicates inside retrieval whenever the store supports it. Post-filtering a small top-k leaves you with too few legal results some of the time, and it drags forbidden text through application memory and logs on the way.

Query rewriting can help either branch, and it can quietly destroy the signal. Expanding "SSO loop" into a fuller diagnostic query might surface prose that says repeated authentication redirect, which is exactly what you wanted. Swapping the user's literal error code for a paraphrase deletes the single best BM25 term you had. Keep the original query, search the rewritten variants alongside it, and evaluate the fusion, instead of assuming the more articulate query is the better one.

Embedding search uses a bi-encoder: the query and the passage are encoded separately, which is what lets you precompute every passage vector ahead of time. That's why search over millions of candidates is fast. The price is that the query and the passage never get to inspect each other's tokens while being scored^[7].

A cross-encoder reads the query and one candidate together. Full token-level attention buys a much sharper relevance judgment at much higher cost. So the standard cascade retrieves tens or hundreds of candidates cheaply, reranks that short list, and sends only the best handful into the prompt.

The cost gap is about reuse. A bi-encoder computes each passage vector once, during ingestion, and does one cheap comparison per request. A cross-encoder has to run a fresh joint forward pass for every query-passage pair it sees. Reranking 100 candidates is roughly 100 pair evaluations, though batching keeps the hardware busy. Point that model at a million passages per query and you've thrown away the entire reason the index exists.

Rerankers are also where you can enforce the policies vector similarity has no clean way to express: source authority, recency, document type, diversity, or whether a candidate carries enough context to answer at all. Keep the learned relevance separate from the explicit business rules, so you can inspect each one on its own when the ordering surprises you.

Diversity starts to matter the moment your first page is five overlapping chunks from the same document. Sending all five spends budget on repeated sentences and can crowd out the second source that would have confirmed or contradicted the first. Deduplicating by source offset, maximal marginal relevance, or just a per-document cap will often improve the evidence packet even as the raw relevance scores tick down.

Assembly is where ranked chunks become a bounded evidence packet. Reserve tokens for the system instructions, the conversation history, the tool schemas, the user's question, and the model's answer. Whatever's left is the retrieval budget. Filling that space to the brim can make the answer worse, not better, by introducing conflicts and distractions the model then has to resolve.

Order matters too. Models tend to underuse evidence buried in the middle of a long context — the lost-in-the-middle effect. Put the strongest material wherever your chosen model reliably reads it, group related chunks together, drop the duplicates, and keep source boundaries visible. Then tell the model how to cite, what to do when two sources conflict, and when to abstain because the evidence isn't there.

Liu and collaborators measured exactly this in 2023^[8] by sliding the relevant information through long inputs and watching what happened. Performance was usually strongest when the evidence sat near the beginning or the end, and weakest in the middle, even for models whose context window covered the whole input comfortably. A context window tells you how many tokens the model will accept. It promises nothing about equal attention across every position.

Compression is another model call, which means another failure surface. Extractive compression keeps selected source sentences and holds onto provenance cleanly. Abstractive summaries save more tokens and can quietly erase a qualification or invent a connective claim that was never in the source. So evaluate the compressed evidence itself, not just the prose that comes out the other end.

Citations need a mechanical path back to chunks. Give each context block a stable source label, instruct the model to attach those labels to its claims, and then, after generation, check that every label it cited was actually in the supplied context and resolve it to a document location. That catches fabricated identifiers and broken links. Whether the cited passage genuinely supports the sentence is a separate question, and it needs a faithfulness judgment rather than a lookup.

Multi-hop questions make packing harder, because no single passage holds the answer. One chunk names the team that owns a product; another gives that team's escalation policy. A single retrieval pass can easily miss the bridge between the two. Query decomposition or iterative retrieval can find each step, but every extra round adds latency and gives an early mistake another chance to steer the later searches.

When an answer is wrong, the first question is whether the evidence was even in the corpus. Then, in order: did ingestion parse it, did retrieval find it, did reranking keep it, did assembly include it, and did generation use it? A single end-to-end score answers none of those, which is why it can't tell you what to fix.

That trace doubles as a debugging order. Start from the source of truth and the passage you expected, then inspect each artifact downstream of it. If exact search can't find the expected chunk, the problem is parsing or chunking. If exact search finds it but approximate nearest-neighbor search doesn't, the index needs tuning. If retrieval succeeds and the passage disappears somewhere in reranking or assembly, the generator had nothing to do with that miss.

Retrieval metrics need labeled relevance to mean anything. Hit rate@k asks whether at least one relevant item showed up. Recall@k asks how much of the relevant set showed up. MRR rewards putting the first relevant result near the top. nDCG handles graded relevance and position together. Track latency and filtered-result counts next to all of them.

Generation metrics have to keep faithfulness — are the claims supported by the supplied context? — apart from answer relevance — does the response address what was asked? Context relevance is the third leg, measuring whether the retrieved material was useful rather than merely on-topic. Together this RAG triad catches both the fluent answer that ignores its evidence and the perfectly faithful answer built from evidence that was beside the point.

Build a small human-reviewed evaluation set before you start tuning anything. Put answerable and unanswerable questions in it, exact identifiers and paraphrases, permission boundaries, stale documents, conflicting sources, tables, and multi-hop cases. Model-based judges can scale the review once it exists, as long as you calibrate them against humans and keep the examples behind every aggregate score.

Evaluation sets age along with the corpus. A policy question can quietly change its correct passage after a revision, and a question that was unanswerable last month becomes answerable the day the right document lands. Store the expected source versions with the relevance labels, review the disagreements, and keep a stable regression slice next to a rotating sample of current traffic. Production feedback helps, with the caveat that clicks and thumbs-up measure presentation and user expectation at least as much as retrieval quality.

Abstention needs tests of its own. Include questions whose answer isn't in the corpus, questions whose sources contradict each other, and questions whose only matching document sits outside the user's permissions. Score whether the system declines cleanly, and whether its explanation manages to avoid leaking the restricted detail it just refused to give. An answer-quality average will happily conceal a system that does well on easy questions and improvises dangerously the moment the evidence runs out.

Use plain long context when the source set is small, already selected, and fits without a fight. A contract reviewer working through one uploaded agreement needs careful parsing and prompting far more than a persistent retrieval index. Search only adds failure modes when there's nothing meaningful to search.

Use tools when the task needs an action or an exact live value. A database query, a calculator, a filesystem read, an API call — those return authoritative structured results. An agent can still retrieve documentation alongside them. What it shouldn't do is let vector search impersonate a transactional system.

Use fine-tuning when what you want is behavior: a stable tone, a fixed format, a classification policy, a procedure repeated the same way every time. Retrieval can supply examples and facts, and cramming a pile of demonstrations into every prompt usually costs more and behaves less consistently than a tuned model would.

Use a direct structured lookup when the question has exactly one authoritative row behind it. Account balance, shipment status, permission checks — those belong behind an API or a database query with typed results. Retrieval can find the policy that explains a balance while the live amount comes from the transactional source. Both can appear in one answer, as long as nobody pretends they carry the same guarantee.

None of these choices excludes the others. A support agent can retrieve the policy, call the order API, and answer in a fine-tuned format, all in one turn. What should drive the architecture is the information source and the guarantee each step actually needs.

The working pipeline ends where it started, with evidence. Keep the sources current. Make retrieval measurable. Carry provenance through prompt assembly. Hold the generator inside the record it was given, and expose every handoff in the trace. Do that and a wrong answer stops being a mystery and becomes an engineering problem with an address.