The RAG Pipeline, Stage by Stage
Every stage of the indexing and query pipelines — what it does, what it outputs, what it costs, and how it fails
The RAG Pipeline, Stage by Stage
TL;DR
A RAG system is two pipelines. The indexing pipeline runs offline and turns documents into searchable chunks. The query pipeline runs on every question and turns a question into a grounded answer. This page walks every stage of both, naming what goes in, what comes out, and how each one fails.
| Property | Value |
|---|---|
| Level | Beginner → Intermediate |
| Reading time | ~25 minutes |
| Prerequisites | What Is RAG? |
| You will understand | The full data flow, and where in it any given bug lives |
The Whole Machine
Complete RAG architecture
Sources
Indexing — offline
Storage
Query — online
Delivery
Read that diagram top to bottom once. The middle layer — Storage — is the boundary between the two pipelines. Indexing writes to it; querying reads from it. They never talk to each other directly.
Part 1 — The Indexing Pipeline
Runs when documents arrive or change. Nobody is waiting, so correctness matters far more than speed.
Stage 1 — Load
Input: files, URLs, database rows, API responses. Output: raw text plus metadata.
Turning a source into text is rarely trivial. A PDF is not a text file — it is a set of drawing instructions that happen to place glyphs on a page. Extracting reading order, tables, and headings from it is genuine work.
| Source | Difficulty | The trap |
|---|---|---|
| Plain text, Markdown | Easy | Almost none |
| HTML | Medium | Navigation, ads, and footers pollute the text |
| Word documents | Medium | Tracked changes and comments leak in |
| Digital PDF | Hard | Multi-column layouts get interleaved into nonsense |
| Scanned PDF | Very hard | It is an image — needs Optical Character Recognition (OCR) |
| Slides | Very hard | Meaning lives in layout and images, not sentences |
Garbage in, garbage forever. If your loader interleaves a two-column PDF into scrambled sentences, every later stage inherits the damage. No amount of clever retrieval repairs a broken parse. Always read a sample of your extracted text with your own eyes before building anything on top of it.
Covered fully in Document Loading & Parsing.
Stage 2 — Clean
Input: raw extracted text. Output: normalised text with noise removed.
Typical operations: strip repeated headers and footers, drop page numbers, collapse runaway whitespace, fix encoding artefacts (’ → '), remove boilerplate that appears on every page, and normalise Unicode.
The payoff is direct: repeated boilerplate creates chunks that look similar to everything, so they surface for unrelated queries and crowd out real answers.
Stage 3 — Chunk
Input: clean document text. Output: a list of chunks, each a few hundred tokens.
This is the single highest-leverage decision in the whole system.
Why split at all? Three reasons:
- Precision. Retrieving a 40-page document to answer one question buries the answer in noise.
- Cost. You pay per token. Sending only the relevant paragraph is dramatically cheaper.
- Limits. Embedding models have their own input caps — often 512 to 8,192 tokens.
The defining risk: a chunk boundary that lands in the middle of a fact destroys that fact. Suppose chunk 7 ends with "the maximum dose is" and chunk 8 begins with "40 mg daily". Neither chunk answers "what is the maximum dose?" Retrieval will never find that answer, because the answer no longer exists anywhere.
Covered fully in Chunking Strategies.
Stage 4 — Enrich
Input: bare chunks. Output: chunks with metadata and added context.
Two distinct enrichments, often confused:
Metadata — structured fields stored alongside the chunk: source file, page number, section heading, author, date, department, access level. Metadata is what makes filtered search possible ("only 2025 documents", "only documents this user may read"). Access-control metadata in particular is not optional in any multi-user system.
Contextualisation — text prepended to the chunk itself so it stands alone. A chunk reading "This applies only to patients over 65" is useless on its own — what applies? Add the section heading and a one-line summary of the parent document to the front. The chunk now stands alone, and retrieval measurably improves.
Before: "This applies only to patients over 65."
After: "[Anticoagulation Protocol 2025 > Section 4: Dose Adjustment]
This applies only to patients over 65."Stage 5 — Embed
Input: chunk text. Output: a vector — a fixed-length list of numbers, typically 384 to 3,072 of them.
An embedding model reads text and produces coordinates in a "meaning space" where texts about similar things land near each other. This is what makes searching by meaning rather than by keyword possible.
The same embedding model must be used for indexing and for querying. Different models produce coordinates in incompatible spaces — the numbers still compute, but the similarity scores are meaningless noise. If you ever change embedding models, you must re-embed the entire corpus. Budget for this before you choose a model.
Covered fully in Embeddings Explained.
Stage 6 — Store
Input: vectors + chunk text + metadata. Output: a queryable index.
You are usually building two indexes over the same chunks:
- A vector index for meaning-based search
- A keyword index for exact-term search (names, codes, identifiers)
Keeping both is what enables hybrid search, and it is nearly always worth the storage.
Covered fully in Vector Databases & Indexes.
Indexing Pipeline Summary
Indexing — runs offline
Part 2 — The Query Pipeline
Runs on every question. A human is waiting, so latency is a hard constraint.
Stage 1 — Understand the Query
Input: whatever the user typed. Output: one or more clean, searchable queries.
Raw user input is frequently unsearchable, and this stage is skipped far too often.
| Problem | Example | Fix |
|---|---|---|
| Pronouns and references | "Can I take it with that?" | Query rewriting — use conversation history to resolve to "Can I take ibuprofen with warfarin?" |
| Too vague | "Tell me about the protocol" | Query expansion — generate several specific variants |
| Multiple questions at once | "What is the dose and what are the side effects?" | Decomposition — split into separate retrievals |
| Wrong vocabulary | User says "blood thinner", documents say "anticoagulant" | Expansion or HyDE — bridge the vocabulary gap |
| Needs no retrieval at all | "Thanks!" | Routing — skip retrieval entirely |
In a conversational interface, query rewriting is not optional. Without it, the second turn of every conversation retrieves garbage, because "what about the elderly?" contains almost no searchable content on its own.
Stage 2 — Retrieve
Input: the processed query. Output: a candidate set — typically 20 to 100 chunks.
Three retrieval methods, used alone or together:
| Method | Matches on | Strong at | Blind to |
|---|---|---|---|
| Dense / vector | Meaning | Paraphrases, synonyms, concepts | Exact codes, rare names, identifiers |
| Sparse / keyword (BM25) | Literal words | Names, codes, acronyms, exact quotes | Synonyms and rephrasing |
| Hybrid | Both, fused | Nearly everything | Very little — this is the default in production |
Retrieve wide, then narrow
Retrieve far more candidates than you intend to use — 20 to 100 — because the next stage is much better at judging relevance than this one. This is the recall-then-precision pattern, and it underpins essentially every serious RAG system.
Covered fully in Retrieval Strategies.
Stage 3 — Rerank
Input: 20–100 candidates. Output: the best 3–8, in order.
The retrieval stage compares the question and each chunk separately — it embedded them independently, possibly weeks apart. A reranker reads the question and chunk together and scores their actual relationship. That is far more accurate, and far too slow to run over the whole corpus — which is exactly why it runs second, over a small candidate set.
Covered fully in Reranking & Context Assembly.
Stage 4 — Assemble the Context
Input: the top chunks. Output: one finished prompt.
Not merely concatenation. This stage decides:
- Order — models attend most reliably to the start and end of a context; put the strongest chunk first
- Budget — how many tokens context may consume
- Deduplication — overlapping chunks repeat text; repetition wastes budget and can bias the model
- Attribution markers — the
[1],[2]labels that make citation possible - Instructions — the grounding rules that make the model behave
Stage 5 — Generate
Input: the assembled prompt. Output: the answer text.
The model writes the answer. This stage dominates latency — typically 70–90% of total response time — which is why the earlier stages can afford to be thorough.
Covered fully in Generation & Grounding.
Stage 6 — Post-process
Input: raw model output. Output: what the user actually sees.
Attach citations, verify that claims trace to the supplied context, apply safety filters, format, stream, and log everything for later evaluation.
Logging is not an afterthought. You cannot improve a RAG system you cannot measure, and you cannot measure one that did not record which chunks it retrieved. Log the query, the retrieved chunk IDs, their scores, and the final answer from day one.
Query Pipeline Summary
Query — runs per question
Where Time and Money Go
Rough figures for a typical production system. Exact numbers vary widely, but the proportions are stable and worth internalising.
| Stage | Typical latency | Share of total | Notes |
|---|---|---|---|
| Query rewriting | 100–300 ms | ~10% | A small model call; skippable for single-turn |
| Embedding the query | 20–60 ms | ~3% | One short input; very cheap |
| Vector search | 10–50 ms | ~3% | Sub-linear thanks to the index |
| Keyword search | 5–30 ms | ~2% | Runs in parallel with vector search |
| Reranking | 50–200 ms | ~8% | Scales with candidate count |
| Generation | 800–3,000 ms | ~75% | Dominates everything |
Two consequences follow directly from that table. First, optimising retrieval for speed is usually wasted effort — it is already a rounding error next to generation. Second, streaming the answer is the highest-impact latency work you can do, because it changes when the user sees the first token rather than how long the whole thing takes.
Where Bugs Live
When a RAG system gives a bad answer, the cause is almost always locatable to one stage. Work through this in order.
| Symptom | Most likely stage | How to confirm |
|---|---|---|
| Answer invents facts not in any document | Generation | Read the retrieved chunks — was the fact there? If not, it is a retrieval failure wearing a generation costume |
| Answer says "I don't know" but the fact exists | Retrieval | Search the index directly for the fact. Present but not retrieved → retrieval. Absent → chunking or loading |
| Answer is half right, missing a detail | Chunking | The fact was probably split across a boundary |
| Right document, wrong section | Chunking or reranking | Chunks too large, or the reranker is misordering |
| Fails only on names, codes, IDs | Retrieval | Vector-only search. Add keyword search |
| Fails only on the 2nd+ conversation turn | Query understanding | No query rewriting — the follow-up has no searchable content |
| Text retrieved is garbled | Loading | Read the raw extracted text directly |
| Was fine, degraded over time | Storage / freshness | Stale index, or embedding model changed underneath you |
The single most useful debugging habit
Always look at the retrieved chunks before blaming the model. Print them. Read them. A large majority of "the LLM hallucinated" reports turn out to be "the right chunk was never retrieved." Diagnosing generation before you have inspected retrieval wastes enormous amounts of time.
Covered fully in Failure Modes & Debugging.
Concept Checks
Check yourself
Why is the storage layer the boundary between the two pipelines?
Because it is the only thing they share. Indexing writes chunks, vectors, and metadata into storage; querying reads from it. They run at different times, at different speeds, triggered by different events. This decoupling is what lets you re-index without any downtime for queries.
Your system answers turn 1 well and turn 2 badly. What is broken?
Query understanding — specifically, missing query rewriting. Turn 2 is typically something like "and for elderly patients?", which contains almost no retrievable content on its own. Without rewriting it against the conversation history, you are searching the index for a fragment.
Why retrieve 50 candidates just to use 4?
Because the two stages have different strengths. Vector search is fast but approximate — it is good at not missing things (recall) and mediocre at ordering them. Reranking is accurate but slow, so it can only run on a small set. Retrieving wide then reranking narrow gets you the recall of the first and the precision of the second.
You cut vector search latency from 40 ms to 10 ms. What did the user notice?
Essentially nothing. Generation is ~75% of a ~2-second response, so you improved the total by roughly 1.5%. That engineering effort would have paid off far better spent on retrieval quality, or on streaming the response.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Two pipelines | Indexing (offline, per document) and query (online, per question) |
| Storage boundary | The only shared surface; enables re-indexing without downtime |
| Load | Source → text. Broken parsing is unrecoverable downstream |
| Chunk | Split into retrievable units. Boundaries destroy facts that cross them |
| Enrich | Metadata for filtering; context prefixes for standalone meaning |
| Embed | Text → vector. Same model for index and query, always |
| Retrieve wide, rerank narrow | Recall from search, precision from the reranker |
| Generation dominates latency | ~75% of response time; optimise quality elsewhere |
| Debug retrieval first | Print the chunks before blaming the model |
Next
Start at the beginning of the pipeline: Document Loading & Parsing.