Designing a RAG System
A complete worked example — gathering requirements, sizing the index, choosing every component with reasons, and planning the rollout
Designing a RAG System
TL;DR
This page puts everything together. We take one realistic brief, ask the questions that actually shape the design, do the arithmetic, and choose every component with a reason attached. Then we change one requirement and watch half the decisions flip — because there is no universally correct RAG design, only a design that fits a set of constraints.
| Property | Value |
|---|---|
| Level | Advanced — brings together every previous page |
| Reading time | ~30 minutes |
| Prerequisites | Failure Modes & Debugging |
| You will understand | How to go from a vague brief to a sized, justified design |
Part 1 — Ask Before You Build
Most bad RAG designs come from starting with the technology. Start with these questions instead.
| Question | Why it changes the design |
|---|---|
| Who asks the questions? | Staff will tolerate a slower, more careful system. Customers will not. |
| What does a wrong answer cost? | A wrong recipe suggestion is a shrug. A wrong drug interaction is a lawsuit. This sets how much verification you can afford. |
| How many documents, and how big? | Decides storage, index type, and whether you need more than one machine |
| How often do documents change? | Daily changes need incremental indexing. Yearly changes do not. |
| How many questions per day? | Sets your cost and your capacity |
| How fast must an answer arrive? | Under 1 second rules out reranking plus multi-step reasoning |
| Can everyone see everything? | Per-user permissions change the whole retrieval design |
| What is the budget? | Turns "nice to have" into "pick two" |
| What does success look like? | If nobody can define it, you cannot evaluate it |
The two questions that change a design most are "what does a wrong answer cost?" and "can everyone see everything?". Everything else adjusts sizes and budgets. Those two change the architecture.
Part 2 — The Brief
We are a software company. We want an assistant that answers customer support questions from our help articles, product manuals, and past support tickets. Customers use it on our website. We have about 50,000 documents. We expect around 200,000 questions a month. It must feel fast. Wrong answers embarrass us but nobody dies. Budget is modest.
Turning that into numbers:
| Requirement | Value |
|---|---|
| Users | Public customers, not staff |
| Corpus | ~50,000 documents, averaging ~8 pages |
| Change rate | Help articles change weekly; tickets arrive constantly |
| Volume | 200,000 questions/month |
| Latency target | First words on screen in under 1 second |
| Cost of a wrong answer | Moderate — reputation, not safety |
| Permissions | Public content only. Everyone sees the same thing. |
| Success measure | Fewer support tickets escalated to humans |
That last row is the valuable one. "Fewer escalations" is countable, tied to money, and gives you a real target to evaluate against.
Part 3 — Sizing the Index
Do this arithmetic early. It decides several choices at once and takes five minutes.
DOCUMENTS
50,000 documents × 8 pages × ~500 words = 200,000,000 words
≈ 267,000,000 tokens (roughly 1.33 tokens per word)
CHUNKS
500 tokens per chunk, 50 tokens overlap
→ ~450 new tokens per chunk
267,000,000 ÷ 450 ≈ 594,000 chunks
Round up for tickets and metadata ≈ 650,000 chunks
VECTOR STORAGE (1024 dimensions, 4-byte floats)
650,000 × 1024 × 4 bytes ≈ 2.7 GB raw
+ index overhead (HNSW, roughly 1.5×) ≈ 4.0 GB
With int8 quantization instead ≈ 1.0 GB
TRAFFIC
200,000 ÷ 30 days ≈ 6,700 per day
Spread over ~10 busy hours ≈ 0.2 per second average
Assume a 10× peak ≈ 2 per second peakTwo conclusions fall straight out of this. First, 4 GB fits comfortably in memory on one ordinary machine — you do not need a distributed setup, and anyone proposing one is overbuilding. Second, 2 queries per second is a very light load. Your constraints here are cost and quality, not scale.
Part 4 — The Decisions
Each choice below names the requirement that drove it.
Ingestion and chunking
| Decision | Choice | Driven by |
|---|---|---|
| Parsing | Layout-aware parser for manuals; plain text for tickets | Manuals have tables and columns; tickets are plain |
| Chunk size | ~500 tokens | Support answers are short and specific; small chunks keep precision high |
| Overlap | ~50 tokens | Cheap protection against a fact splitting across a boundary |
| Splitting | Recursive, on headings and paragraphs | Help articles are already well structured — use that structure |
| Enrichment | Prepend the article title and section heading to each chunk | Makes each chunk understandable on its own |
| Metadata | Product, version, date, type, URL | Needed for filtering, recency, and citation links |
Retrieval
| Decision | Choice | Driven by |
|---|---|---|
| Search | Hybrid — vector + BM25, fused with RRF | Support questions contain error codes and version numbers. Vector search alone would miss them. |
| Embedding model | A mid-size model, 1024 dimensions, self-hosted | Volume is high enough that per-call embedding fees add up; quality at this size is good |
| Index | HNSW with int8 quantization | Fits in memory, fast, and quantization cuts RAM by roughly 4× at a small recall cost |
| Initial k | 30 candidates | Wide enough for the reranker to have real choice |
| Reranking | Cross-encoder, keep the top 4 | Biggest single quality gain available, and it reduces generation cost |
| Filtering | Pre-filter by product and version | A customer on v2 must not get v3 instructions |
That last row is easy to miss and causes real support failures. Two documents can both be correct and yet one is wrong for this user. Version and product must be filters, not hints in the text.
Generation
| Decision | Choice | Driven by |
|---|---|---|
| Model | Mid-tier, with routing to a small model for easy questions | Cost. Most support questions are simple lookups. |
| Temperature | 0.1 | Answers should mirror the documentation, not improvise |
| Prompt | Strict grounding, numbered sources, exact refusal wording | Wrong answers are the main risk; refusing is better than inventing |
| Streaming | Yes | The "feels fast" requirement. Generation is most of the wait. |
| Citations | Link to the help article URL | Lets customers self-serve, and builds trust |
Operations
| Decision | Choice | Driven by |
|---|---|---|
| Indexing | Incremental, triggered by article updates; nightly batch for tickets | Articles change weekly; tickets arrive constantly but are not urgent |
| Caching | Exact-match, plus semantic with a high threshold | Support traffic repeats heavily — the same twenty questions dominate |
| Cache keys | Query + product + version | Prevents serving a v2 answer to a v3 customer |
| Escalation | On refusal or low retrieval score, offer a human | Directly serves the success measure |
| Monitoring | Refusal rate, escalation rate, reformulation rate, p95 latency | The metrics that map to the business goal |
The resulting pipeline
Final design
Part 5 — Build It in Stages
Do not build the diagram above on day one. Build the smallest version, measure it, and add only what the measurements justify.
Staged rollout
Stage 1 — Naive RAG
Parse, chunk, embed, vector search, generate. Build the golden dataset at the same time.
Stage 2 — Measure
Recall, precision, faithfulness. Now you know what is actually broken.
Stage 3 — Add hybrid search
Almost certainly needed, because of error codes and version numbers
Stage 4 — Add reranking
The largest quality gain per unit of effort
Stage 5 — Add caching and routing
Now optimize cost, once quality is acceptable
Stage 6 — Ship to a small group
Watch real traffic; rebuild the test set from real questions
Stage 1 exists to produce measurements, not a product. Teams that skip it end up with an elaborate pipeline and no idea which part is carrying the quality — so they cannot remove anything, and every change is a guess.
Part 6 — Change One Requirement
Now swap a single line of the brief. The assistant is for hospital clinicians, answering from clinical protocols. A wrong answer can harm a patient.
Corpus size, traffic, and storage are unchanged. Look at how much of the design flips anyway:
| Decision | Support version | Clinical version | Why it changed |
|---|---|---|---|
| Latency target | Under 1 second | Several seconds is fine | Clinicians will wait for a careful answer |
| Streaming before verification | Yes | No | Never show unverified clinical text, even briefly |
| Refusal behaviour | Offer a human | Refuse readily and loudly | Silence is far safer than a wrong answer |
| Semantic caching | Yes | No | "30 mg" and "300 mg" are similar queries with lethally different answers |
| Model | Mid-tier with routing | Strongest available, no routing | Do not economize where the downside is harm |
| Verification | Check citations | Claim-by-claim checking against sources | Justified by the cost of being wrong |
| Permissions | Everyone sees everything | Per-role retrieval filters | Patient data and role boundaries |
| Audit log | Basic | Full — every chunk shown to every user | Regulatory requirement |
| Human in the loop | Optional | Required for anything actionable | Clinical governance |
| Hybrid search | Yes | Yes | Drug names and codes — unchanged |
| Reranking | Yes | Yes | Still the best quality lever — unchanged |
Notice which rows did not change. Chunking, hybrid search, and reranking are the same in both designs. Those are engineering fundamentals that serve every RAG system. What changed is everything downstream of "what does a wrong answer cost?" — caching, streaming, refusals, verification, and oversight.
Concept Checks
Check yourself
Why size the index before choosing a vector database?
Because the arithmetic usually removes the question. 650,000 chunks come to about 4 GB, which fits in memory on one ordinary machine — so distributed systems, sharding, and the databases built for them are all irrelevant. Sizing first turns an open-ended technology debate into a short list. It also exposes the cases where you genuinely do need heavy infrastructure, which is far rarer than people expect.
Why is semantic caching acceptable for support but not for clinical use?
Because it deliberately answers one question with the answer to a similar one. In support, a near-miss produces a slightly-off article and a mildly annoyed customer. In clinical use, "30 mg" and "300 mg" embed almost identically while having opposite safe answers, so the same mechanism can produce a dosing error. The technique did not change; the cost of its failure mode did.
Why does 'fewer escalated tickets' make a better success measure than 'user satisfaction'?
Because it is counted automatically for every interaction, and it maps directly onto money. Satisfaction scores depend on users choosing to respond, which a small and unrepresentative group does, and they drift with mood and wording. Escalations are recorded whether or not anyone opts in, they are unambiguous, and improving them has a value the business already knows how to price.
Why build naive RAG first when you already know you need hybrid search?
Because Stage 1 produces the baseline that makes every later decision checkable. Without it you cannot show that hybrid search helped, cannot tell how much reranking is worth, and cannot spot the change that quietly made things worse. It also forces the golden dataset into existence early. Building the full pipeline first leaves you unable to remove anything, since nothing was ever measured on its own.
Which parts of a RAG design are basically constant across use cases?
Careful parsing, sensible chunking, hybrid search, and reranking. These serve almost every system because they address problems every corpus has — mangled documents, boundaries splitting facts, exact terms that embeddings miss, and rough initial ranking. What varies is the layer above: caching policy, refusal behaviour, verification depth, permissions, and oversight, all driven by what a wrong answer costs.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Start with questions | Requirements shape the design; technology choices follow |
| The two decisive questions | What does a wrong answer cost, and can everyone see everything |
| Size early | Five minutes of arithmetic removes most infrastructure debates |
| Attach a reason to every choice | A design you cannot justify is a design you cannot revise |
| Filter on product and version | Correct-but-wrong-for-this-user is a real failure mode |
| Cache keys carry context | Query alone is not a safe key |
| Build in stages | Stage 1 exists to produce measurements |
| Success must be countable | Escalation rate beats satisfaction surveys |
| Constant across designs | Parsing, chunking, hybrid search, reranking |
| Varies with risk | Caching, streaming, refusals, verification, oversight |
Next
That completes the track. Keep the Glossary open as a reference, then go and build: Intelligent Document Q&A.
Failure Modes & Debugging
A systematic method for diagnosing bad answers — symptom to root cause to fix, across parsing, chunking, embedding, retrieval, reranking, and generation
Glossary
Every RAG term and abbreviation, expanded and explained in one place — from ANN and BM25 to nDCG, RRF, and reflection tokens