RAG Crash Course
All of RAG on one page — what it is, how every stage works, the decisions that matter, how to measure it, and how to fix it when it breaks
RAG Crash Course
Start here
This single page covers all of RAG at working depth. Read it start to finish in about 30 minutes and you will understand what RAG is, how every piece works, which decisions actually matter, and what to do when it goes wrong.
Every section ends with a Go deeper link to a full page on that topic. Read this first, then follow the links for whatever you need in detail.
| Property | Value |
|---|---|
| Level | Everyone — starts from zero, ends at production |
| Reading time | ~30 minutes |
| Prerequisites | None. If you know what a chatbot is, you can read this. |
| You will understand | The whole of RAG, well enough to design, build, and debug one |
1. What RAG Is
RAG stands for Retrieval-Augmented Generation. Three words, and each one is a step:
| Word | Meaning |
|---|---|
| Retrieval | Search your own documents for passages related to the question |
| Augmented | Paste those passages into the prompt |
| Generation | Ask the model to answer using only that text |
That is the entire idea. The model brings language skill. Your documents bring the facts.
RAG in one line
Why it exists
Large Language Models (LLMs) have four problems that RAG fixes:
| Problem | What it looks like | How RAG fixes it |
|---|---|---|
| It does not know your data | It has never seen your contracts, tickets, or wiki | You give it the relevant passage at question time |
| Knowledge cutoff | Its training stopped on some date. It does not know what happened after. | Your documents can be updated today |
| Hallucination | It invents facts, fluently and confidently | It answers from text you supplied, and cites it |
| No sources | You cannot check where an answer came from | Every claim points at a document |
The alternative to RAG is usually fine-tuning, and people mix them up. Fine-tuning changes the model's weights and teaches it style, format, and skill. RAG supplies facts at question time. They solve different problems, and most real systems use both. If your knowledge changes weekly, fine-tuning is the wrong tool — you would have to retrain every week.
Go deeper: What Is RAG? — the full comparison against fine-tuning, long context, and web search, and when not to use RAG.
2. The Two Pipelines
This is the idea beginners most often miss. A RAG system is two separate pipelines that never talk to each other directly.
Two pipelines, one shared storage
Indexing (offline)
Runs ahead of time, once per document. Load → clean → chunk → embed → store. Slow is fine; nobody is waiting.
Querying (online)
Runs on every question, while a user waits. Search → rerank → build prompt → generate.
The indexing pipeline writes to storage. The query pipeline reads from it. Keeping them separate in your head makes every later concept easier.
Go deeper: The RAG Pipeline, Stage by Stage.
3. One Question, End to End
Press play and watch a real question travel the whole query pipeline:
The query pipeline, step by step
Question
+0 msThe user asks something in plain language.
Data leaving this stage
"Can I take ibuprofen with warfarin?"
Nothing has happened yet. This raw string is all the system has to work with.
Look at the timing bar: generation dominates everything else. Retrieval and reranking together cost less than a fifth of the total, which is why adding a reranker is usually a cheap upgrade in latency terms.
Note where the time goes. Generation is most of it. Search is almost free. That surprises people, and it decides where optimizing is worth your effort.
4. Every Stage, and the One Thing That Matters
Indexing — building the searchable copy
Stage 1: Load and parse
Turn PDFs, HTML, Word files, and scans into plain text.
The one thing that matters: a PDF is not a text file — it is a page-drawing program. Tables get flattened into meaningless number soup. Two-column pages come out scrambled. Scans contain no text at all, only a picture of text.
Read your parsed text with your own eyes, across every file type you handle. Parsing failures are silent. Nothing errors, documents index fine, and the system simply answers as though half your content does not exist.
Go deeper: Document Loading & Parsing.
Stage 2: Chunk
Split documents into small pieces. Each piece is what gets stored, searched, and pasted into the prompt.
The one thing that matters: a boundary landing in the middle of a fact destroys that fact permanently.
Chunk 7 ends: "...the maximum dose is"
Chunk 8 begins: "40 mg daily..."
Question: "What is the maximum dose?"
Answer: Nothing. Neither chunk contains it.
No prompt, model, or reranker can recover it.| Setting | Sensible default | Why |
|---|---|---|
| Chunk size | 300–800 tokens | Small enough to be precise, big enough to make sense alone |
| Overlap | 10–15% | Cheap insurance against splitting a fact |
| Split on | Headings, then paragraphs, then sentences | Follow the document's own structure |
| Add to each chunk | Its document title and section heading | Makes the chunk understandable on its own |
Chunking visualizer
The same clinical note, split three ways. Watch what happens to the sentence explaining the drug interaction as you shrink the chunks.
Cuts at the nearest separator (paragraph, then sentence, then word) before the limit. The sensible default.
Chunks
5
Avg size
~29
tokens
Storage cost
121%
of original
The key fact survived
One chunk contains the full phrase "warfarin and ibuprofen", so a search for the drug interaction can retrieve it.
Patient was admitted on March 3 with shortness of breath. Chest imaging showed bilateral infiltrates. She was started on warfarin 5 mg daily for atrial fibrillation.
aily for atrial fibrillation. Her INR on admission was 2.4, within the target range of 2.0 to 3.0. On day two the team added ibuprofen for joint pain.
ded ibuprofen for joint pain. By day four the INR had risen to 4.8 and she developed bruising on both arms.
eloped bruising on both arms. The interaction between warfarin and ibuprofen was identified as the likely cause. Ibuprofen was stopped and vitamin K was given.
pped and vitamin K was given. The INR returned to 2.6 within 48 hours and she was discharged on day seven.
Go deeper: Chunking Strategies — semantic, parent-document, and late chunking.
Stage 3: Embed
Turn each chunk into an embedding: a list of numbers that represents its meaning. Texts about similar things get similar numbers.
The one thing that matters: the query and the documents must use the same embedding model. Mix two models and every score becomes meaningless — the system keeps working and returns near-random results.
Cosine similarity is just an angle
Comparing
dog vs puppy
Almost the same direction — near-identical meaning
Notice that only direction matters, never length. That is the whole reason cosine similarity is used for text: a long document and a short sentence about the same topic point the same way, so they score as similar. Notice too that “invoice” and “dog” score close to -1. Real embedding models rarely produce strong negatives like this — most unrelated text lands near 0 — but the geometry is the same.
Go deeper: Embeddings Explained.
Stage 4: Store
Put the vectors in a vector database that can find the closest ones fast.
The one thing that matters: always store the chunk text next to its vector. A vector cannot be turned back into text. Without the text you cannot read what was retrieved — which is the single most useful debugging step in RAG.
| Term | Meaning |
|---|---|
| ANN | Approximate Nearest Neighbour — finds almost the closest vectors, far faster than checking all of them |
| HNSW | The most common ANN index. Fast and accurate, uses a lot of memory |
| IVF | Groups vectors into clusters and searches only the nearest few. Lighter on memory |
| Quantization | Storing vectors at lower precision to cut memory 4–32×, losing a little accuracy |
Go deeper: Vector Databases & Indexes.
Querying — answering the question
Stage 5: Search
Find the chunks most related to the question. There are two ways, and they fail on opposite inputs:
| Method | How it works | Great at | Bad at |
|---|---|---|---|
| Dense (vector) | Compares meaning | Paraphrases — "laptop slow" finds "thermal throttling" | Exact codes like WF-2249 |
| Sparse (BM25) | Compares words | Exact codes, names, rare terms | Paraphrases |
The most useful single fact in this course
Run both and merge the results. This is called hybrid search, and it is the correct default for production. Because the two methods fail on opposite kinds of input, together they cover each other. The standard way to merge them is RRF (Reciprocal Rank Fusion), which combines two ranked lists without needing their scores to be comparable.
Keyword vs vector vs fusion
Three queries, each designed to break one of the two search methods. Compare the top results in each column.
Keyword
BM25 — matches literal words
Protocol WF-2249 defines anticoagulation monitoring for inpatients.
score 9.4
Warfarin interacts with NSAIDs, raising the risk of bleeding.
no match
Blood thinners require regular monitoring of clotting time.
no match
Ibuprofen is a non-steroidal anti-inflammatory drug used for pain.
no match
Vector
Cosine — matches meaning
Protocol WF-2249 defines anticoagulation monitoring for inpatients.
score 0.42
Warfarin interacts with NSAIDs, raising the risk of bleeding.
score 0.39
Blood thinners require regular monitoring of clotting time.
score 0.37
Patients on anticoagulants should report unusual bruising promptly.
score 0.35
Fusion
RRF — combines both ranks
Protocol WF-2249 defines anticoagulation monitoring for inpatients.
rrf 0.0328
Warfarin interacts with NSAIDs, raising the risk of bleeding.
rrf 0.0323
Blood thinners require regular monitoring of clotting time.
rrf 0.0317
Ibuprofen is a non-steroidal anti-inflammatory drug used for pain.
rrf 0.0310
score = 1/(k + keyword_rank) + 1/(k + vector_rank). A small k lets a single first-place finish dominate the fused ranking; a large k flattens the contributions so a document must do well in both lists. 60 is the common default.
Keyword search nails this instantly — the code is a literal string match. Vector search is nearly blind: to an embedding model, "WF-2249" is meaningless noise, so every document looks equally unrelated.
Go deeper: Retrieval Strategies — plus query rewriting, expansion, and HyDE.
Stage 6: Rerank
Search is fast but rough, because it compared the question and each chunk separately. A reranker reads them together and scores the real relationship. Much more accurate, far too slow to run over everything.
So: retrieve wide, rerank narrow. Fetch 20–50 candidates, rerank, keep the best 3–5.
Reranking is the best value change in all of RAG. It adds around 15% to your response time and often makes queries cheaper, because a tighter context means fewer input tokens to pay for.
Go deeper: Reranking & Context Assembly.
Stage 7: Build the prompt
Assemble the surviving chunks, the question, and the rules.
You are a support assistant for Acme.
Rules:
- Answer using ONLY the context below.
- If the context does not contain the answer, reply exactly:
"I don't have that information in my sources."
- Cite the source number in brackets after each claim.
- If sources disagree, say so and show both.
Context:
[1] (refund-policy.pdf, p.2) Hardware may be returned within 30 days...
[2] (refund-policy.pdf, p.3) Software licences are non-refundable once...
Question: Can I return software?The one thing that matters: give the model an exact refusal sentence. A model asked a question will strongly tend to produce an answer — that is what it was trained to do. Unless refusing is made easy and concrete, it will invent something instead.
Order matters too. Models read the start and end of a long context most reliably, and skim the middle. This is called lost in the middle, and it has a surprising consequence: adding more retrieved chunks can make answers worse. Put the strongest chunk first and the question last.
Stage 8: Generate and cite
The model writes the answer from the context.
| Setting | Value | Why |
|---|---|---|
| Temperature | 0–0.3 | You want faithful reproduction of your documents, not creativity |
| Streaming | On | Generation is most of the wait, so showing words as they arrive is the biggest speed win users actually feel |
| Citations | Required | Turns "trust me" into "check me" |
Check the citations in code. You assembled the context, so you know exactly what chunk [2] contains. Models do sometimes cite the wrong number. A confidently wrong citation is worse than none, because it earns trust it does not deserve.
Go deeper: Generation & Grounding.
5. What To Actually Build
Do not build everything. Build this, in this order, measuring at each step:
The order that works
1. Naive RAG
Parse, chunk, embed, search, generate. Build a small test set at the same time.
2. Measure it
Now you know what is actually broken, instead of guessing.
3. Add hybrid search
Almost always needed. Fixes exact codes and names.
4. Add reranking
The biggest quality gain for the least work.
5. Add caching and routing
Optimize cost only after quality is good enough.
The pragmatic stack
Good parsing + sensible chunking + hybrid search + a reranker. That combination is simple, cheap, easy to debug, and solves the large majority of RAG problems. Exhaust it before reaching for anything cleverer.
6. Measuring It
You cannot improve what you cannot measure, and "it looks good when I try it" is not measuring.
Split it in two
A wrong answer has exactly two possible causes, and they share no fixes:
| Cause | Meaning | Where to look |
|---|---|---|
| Retrieval failed | The right text was never in the context | Parsing, chunking, embedding, search |
| Generation failed | The right text was there and the model ignored it | Prompt, context order, model |
Always check which one first. Teams routinely spend weeks tuning prompts for a problem that lives in their chunking.
The metrics
| Metric | Question it answers | When to use it |
|---|---|---|
| Recall@k | Of all relevant chunks, how many did we find? | Before reranking. You cannot rerank what you never retrieved. |
| Precision@k | Of what we returned, how much is relevant? | After reranking. |
| MRR | How high was the first relevant result? | When there is one right answer |
| nDCG@k | Ranking quality, allowing partial relevance | The most informative retrieval metric |
| Faithfulness | Is every claim supported by the context? | The anti-hallucination metric |
| Answer relevance | Does the answer address the question? | Catches fluent, well-sourced, off-topic answers |
Retrieval metrics, calculated live
Search results, best first — click to change relevance
at least one relevant found
Coarsest signal. Easily flattering.
3 found ÷ 5 relevant in corpus
The metric that matters BEFORE reranking.
3 relevant ÷ 5 returned
The metric that matters AFTER reranking.
1 ÷ 1 (rank of first relevant)
Rewards putting the answer first.
DCG 3.76 ÷ ideal DCG 3.76
Uses graded relevance and punishes low ranks.
The best results are at the top, which is what every rank-aware metric rewards. MRR is a perfect 1.0 because the very first result was relevant.
Try the presets in order. The thing to take away is that these numbers disagree with each other on purpose. Recall ignores order entirely; MRR cares about almost nothing except the first hit; nDCG sits in between and is the only one that uses partial relevance. Reporting one number hides whichever failure that number is blind to.
Put unanswerable questions in your test set. If every test question has an answer, a system that always answers scores perfectly — so all your tuning pushes toward hallucination, and correct refusals are never rewarded.
Go deeper: Evaluation & Metrics.
7. When It Breaks
Start with one question. It splits every RAG bug in half:
Was the correct text in the context window?
Check the logged chunk IDs. Answer this before touching anything else.
Common symptoms and their real causes:
| Symptom | Usual cause |
|---|---|
| Product codes and IDs are never found | Vector search only — add hybrid search |
| Answers are half right, missing a condition | Chunk boundary split the fact |
| Answers cite documents that no longer apply | Old chunks not deleted when the document was updated |
| Model states things not in your documents | No grounding instruction in the prompt |
| Model ignores a chunk that was right there | Too many chunks — lost in the middle |
| Results are near-random for everything | Query and documents embedded with different models |
| Works in testing, fails for one customer | A filter is silently excluding results |
| Confident answers when nothing was retrieved | Empty retrieval falls through instead of refusing |
Find the cause of a bad answer
Was the correct text actually in the context window?
Look at the logged chunk IDs for the failing query. This one question splits every RAG bug in half, and the two halves share no fixes. Never skip it.
Every path starts with the same question, and that is deliberate. Once you know whether the right text reached the model, half the possible causes disappear. Debugging a prompt when the real bug is in your chunking is the most common wasted week in RAG work.
Go deeper: Failure Modes & Debugging.
8. Production Essentials
| Area | The thing you must get right |
|---|---|
| Speed | Generation is 70–90% of the wait. Stream tokens. Optimizing search is wasted effort. |
| Cost | Context size is most of the bill. A tighter context is cheaper and usually more accurate. |
| Caching | Huge win on repeated questions. But be careful with semantic caching — "30 days" and "60 days" look similar and have opposite answers. |
| Freshness | When a document changes, delete its old chunks first. Otherwise both versions compete and the stale one sometimes wins. |
| Permissions | Filter during the search, never after. Retrieving text a user may not see and telling the model to ignore it is not access control. |
| Multi-tenancy | Put the tenant in every cache key, or one customer gets another's answers. |
| Security | Anyone who can edit a page you index can plant instructions in it. Control who can write to your corpus. |
| Logging | Log the retrieved chunk IDs for every query. Without this you cannot debug anything. |
| Empty retrieval | Must refuse. Never silently fall back to answering from training data. |
Go deeper: Production & Operations and Designing a RAG System.
9. The Named Architectures
You will hear these names. Each one fixes a specific failure of the basic pipeline, and each one costs more.
| Name | What it does | Use it when |
|---|---|---|
| Naive RAG | Retrieve once, generate once | Always start here |
| HyDE | Writes a fake answer and searches with that instead of the question | Users phrase things very differently from your documents |
| RAG-Fusion | Searches several rewordings and merges the rankings | Results are inconsistent across phrasings |
| Self-RAG | Model grades its own retrieval and output, and may skip retrieval | You want fewer unsupported claims |
| CRAG | Grades retrieval quality and falls back when it failed | The system answers confidently from nothing |
| Adaptive RAG | Routes each question to the cheapest path that works | Usually the best one to add. Can cost less than the baseline. |
| Agentic RAG | An agent searches repeatedly, deciding what to look for next | Questions needing several linked lookups |
| Graph RAG | Searches a graph of entities and relationships | The answer lives between documents, not in any one |
What each architecture actually costs
Retrieve once, generate once. The baseline every project should start from.
Turn the question into a vector
Find the 5 closest chunks
Write the answer from those chunks
Time so far
0 ms
Baseline was 965 ms
Cost so far
0.00 units
Baseline was 1.01
Run all five. Two results surprise most people: reranking makes queries cheaper, not dearer, because it shrinks the context you pay for. And Adaptive RAG beats the baseline on both axes, because it stops you spending a full retrieval and generation on questions that never needed one.
These are not a ladder you climb. A well-tuned Naive RAG beats a badly-tuned Agentic RAG on almost everything, including accuracy — and massively on speed and cost. Add one only when you have measured a failure the simpler system cannot fix.
Go deeper: RAG Architectures.
10. Vocabulary You Need
The words that come up constantly:
| Term | Meaning |
|---|---|
| Chunk | One retrievable piece of a document |
| Embedding / vector | A list of numbers representing meaning |
| Cosine similarity | The angle between two vectors. 1 = same, 0 = unrelated |
| Dense retrieval | Search by meaning |
| Sparse retrieval / BM25 | Search by words |
| Hybrid search | Both together — the production default |
| RRF | Reciprocal Rank Fusion — how you merge two ranked lists |
| ANN | Approximate Nearest Neighbour — fast, slightly inexact vector search |
| HNSW / IVF | The two common ANN index types |
| Bi-encoder | Embeds question and document separately. Fast. Your retriever. |
| Cross-encoder | Reads them together. Accurate, slow. Your reranker. |
| top-k | How many results you keep |
| Grounding | Making retrieved context the only allowed source |
| Hallucination | A fluent, confident, false statement |
| Faithfulness | Whether every claim is supported by the context |
| Recall@k | How much of the relevant material you found |
| Precision@k | How much of what you returned was relevant |
| Lost in the middle | Models skim the middle of long contexts |
| Context window | Maximum tokens a model can read at once |
| Token | The unit models read and bill by — roughly ¾ of a word |
Go deeper: Glossary — every term and abbreviation, expanded.
11. The Twelve Rules
If you remember nothing else:
| # | Rule |
|---|---|
| 1 | Start with Naive RAG. Measure before adding anything. |
| 2 | Read your parsed text. Parsing fails silently and caps everything downstream. |
| 3 | A chunk boundary through a fact destroys that fact. Nothing recovers it. |
| 4 | Same embedding model for queries and documents. Always. |
| 5 | Store the chunk text next to the vector. You cannot reverse an embedding. |
| 6 | Use hybrid search. Dense and sparse fail on opposite inputs. |
| 7 | Retrieve wide, rerank narrow. Best value change available. |
| 8 | Fewer, better chunks beat more chunks. Lost in the middle is real. |
| 9 | Give the model an exact refusal sentence. Otherwise it invents. |
| 10 | Log the retrieved chunk IDs. The one log line that makes debugging possible. |
| 11 | Ask "was the right text in the context?" first. It halves every bug. |
| 12 | Generation is most of your time and money. Optimize there, not in search. |
12. Where To Go Next
You now have the whole picture. The full track goes deeper on each part:
The pipeline, in order
| Page | Covers |
|---|---|
| What Is RAG? | The problem, and RAG vs fine-tuning vs long context |
| The RAG Pipeline | Every stage of both pipelines, and how each fails |
| Document Loading & Parsing | PDFs, tables, scans, metadata |
| Chunking Strategies | Fixed, recursive, semantic, parent-document, late chunking |
| Embeddings Explained | Vectors, cosine similarity, choosing a model |
| Vector Databases & Indexes | HNSW, IVF, quantization, filtering |
| Retrieval Strategies | Dense, sparse, hybrid, BM25, RRF, query rewriting |
| Reranking & Context Assembly | Cross-encoders, ColBERT, lost in the middle |
| Generation & Grounding | Prompts, citations, refusals, verification |
Beyond one question at a time
| Page | Covers |
|---|---|
| Conversational RAG | Follow-up questions, query rewriting, chat history |
| Multimodal RAG | Images, charts, tables, scanned pages |
| RAG Architectures | HyDE, Self-RAG, CRAG, Adaptive, Agentic, Graph, Speculative |
Making it real
| Page | Covers |
|---|---|
| Evaluation & Metrics | Recall, nDCG, faithfulness, test sets, LLM-as-judge |
| Production & Operations | Latency, caching, cost, security, multi-tenancy |
| Failure Modes & Debugging | Symptom to cause, across every stage |
| Designing a RAG System | A full worked design with real numbers |
| Glossary | Every term and abbreviation |
Then build one
| Project | Time |
|---|---|
| Intelligent Document Q&A | ~2 hours |
| Hybrid Search | ~4 hours |
| RAG with Reranking | ~4 hours |
| Production RAG Pipeline | ~3 days |