Evaluation & Metrics
Measuring RAG properly — recall@k, precision@k, MRR, nDCG, faithfulness, answer relevance, golden datasets, LLM-as-judge, and online signals
Evaluation & Metrics
TL;DR
You cannot improve a RAG system you cannot measure, and "it looks good when I try it" is not measurement. Evaluation splits into two separate halves: did retrieval find the right text? and did generation use it faithfully? Measure them separately. A single end-to-end score tells you something is broken without telling you which half to fix.
| Property | Value |
|---|---|
| Level | Intermediate → Advanced |
| Reading time | ~28 minutes |
| Prerequisites | RAG Architectures |
| You will understand | Every standard metric, how to build a test set, and how to run evaluation continuously |
Why Separate Retrieval from Generation
A wrong answer has exactly two possible origins:
Diagnosing a bad answer
Retrieval failed
The correct text was never in the context. No prompt change can fix this — the information was not there.
Generation failed
The correct text was in the context and the model ignored, misread, or contradicted it.
This is the single most important idea on the page. Teams routinely spend weeks tuning prompts against a problem that lives in their chunking. Always establish first whether the answer was retrievable at all — check whether the supporting chunk appeared in the context. If it did not, stop looking at the prompt.
Part 1 — The Golden Dataset
Every metric below needs a labelled test set. This is dull work, cannot be skipped, and is the highest-leverage work in evaluation.
| Field | Purpose |
|---|---|
| Question | Phrased the way real users phrase it — including sloppily |
| Ground-truth answer | What a correct response contains |
| Relevant chunk IDs | Which chunks should be retrieved — this is what makes retrieval metrics computable |
| Metadata | Category, difficulty, source document, date |
How to build one
| Source | Quality | Notes |
|---|---|---|
| Real user queries from logs | Best | Genuinely representative. Start here the moment you have traffic. |
| Domain experts writing Q&A | Very good | Expensive but captures what matters in your domain |
| LLM-generated from your chunks | Usable to bootstrap | Generate questions each chunk answers — gives you chunk labels free. Must be human-reviewed; it skews toward questions that are easy for your existing retrieval. |
| Public benchmarks | Poor for your system | They measure a different corpus. Useful only for comparing models in the abstract. |
How many do you need?
50–100 examples is enough to catch regressions and see large effects — start here rather than waiting for perfection. 200–500 gives stable comparisons between similar configurations. Below ~30, run-to-run noise will exceed the differences you are trying to detect and you will chase ghosts.
Cover deliberately: easy lookups, multi-document questions, questions with no answer in the corpus (to test refusal), ambiguous phrasing, rare entities and product codes, and time-sensitive facts.
Part 2 — Retrieval Metrics
All of these ask one question: did the right chunks come back, and how high did they rank?
Notation: k is how many results you keep. A metric written @k is computed over the top k.
The metrics
| Metric | Question it answers | Cares about order? | Use when |
|---|---|---|---|
| Hit Rate@k | Did any relevant chunk appear in the top k? | No | Quick health check; the coarsest useful signal |
| Recall@k | What fraction of all relevant chunks did we get? | No | The key metric before reranking — you cannot rerank what you failed to retrieve |
| Precision@k | What fraction of returned chunks are relevant? | No | Context is polluted with noise; matters after reranking |
| MRR | How high was the first relevant chunk? | Yes | Questions with one right answer |
| MAP | Average precision across all relevant chunks | Yes | Questions with several relevant chunks |
| nDCG@k | Ranking quality with graded relevance | Yes | Relevance is a scale, not a yes/no |
Worked example
Ten chunks are relevant to a question. You retrieve 5, and positions 2 and 4 are relevant.
| Metric | Calculation | Value |
|---|---|---|
| Hit Rate@5 | At least one relevant chunk present | 1.0 |
| Recall@5 | 2 found ÷ 10 relevant | 0.20 |
| Precision@5 | 2 relevant ÷ 5 returned | 0.40 |
| MRR | First relevant at rank 2 → 1 ÷ 2 | 0.50 |
That row set is instructive: Hit Rate says "perfect", Recall says "we missed 80% of the evidence". Reporting only Hit Rate would hide a serious problem — which is exactly why single-metric dashboards mislead.
Change the results below and watch every metric move. The presets are worth clicking through in order:
Retrieval metrics, calculated live
Search results, best first — click to change relevance
at least one relevant found
Coarsest signal. Easily flattering.
3 found ÷ 5 relevant in corpus
The metric that matters BEFORE reranking.
3 relevant ÷ 5 returned
The metric that matters AFTER reranking.
1 ÷ 1 (rank of first relevant)
Rewards putting the answer first.
DCG 3.76 ÷ ideal DCG 3.76
Uses graded relevance and punishes low ranks.
The best results are at the top, which is what every rank-aware metric rewards. MRR is a perfect 1.0 because the very first result was relevant.
Try the presets in order. The thing to take away is that these numbers disagree with each other on purpose. Recall ignores order entirely; MRR cares about almost nothing except the first hit; nDCG sits in between and is the only one that uses partial relevance. Reporting one number hides whichever failure that number is blind to.
MRR — Mean Reciprocal Rank
For each question: reciprocal rank = 1 / (rank of first relevant result)
MRR = mean of those values across all questions
rank 1 → 1.00 rank 2 → 0.50 rank 3 → 0.33 rank 5 → 0.20The steep drop from rank 1 to rank 3 is deliberate: MRR strongly rewards putting the answer first.
nDCG — Normalized Discounted Cumulative Gain
The most informative retrieval metric, and the one worth understanding properly. Three ideas stacked:
| Part | Meaning |
|---|---|
| Gain | Each result has a relevance score — not just 0/1 but graded: 3 perfect, 2 useful, 1 tangential, 0 irrelevant |
| Discounted | Gain is divided by log2(rank + 1), so a result found at rank 8 counts far less than the same result at rank 1 |
| Normalized | Divided by the score of a perfect ranking, putting the result on a 0–1 scale that is comparable across questions |
DCG@k = Σ relevance_i / log2(i + 1) for i = 1..k
IDCG@k = the same sum over the ideal ranking
nDCG@k = DCG@k / IDCG@k → 1.0 means a perfect rankingUse nDCG when relevance genuinely is graded — a chunk that fully answers the question and one that mentions the topic in passing should not score identically.
Context Precision and Context Recall
These are the RAG-specific framings you will meet in evaluation libraries:
| Metric | Definition |
|---|---|
| Context Recall | Of the claims in the ground-truth answer, how many are supported by the retrieved context? Did we retrieve everything needed? |
| Context Precision | Are the relevant chunks ranked above the irrelevant ones? Is the context signal-dense? |
Choosing k
| Stage | Typical k | Optimize for |
|---|---|---|
| Initial retrieval | 20–100 | Recall. Cast wide; the reranker will clean up. |
| After reranking | 3–8 | Precision. Only what earns its context tokens. |
Measure recall@k at the retrieval stage and precision@k after reranking. Applying the same metric to both stages misjudges each one. Wide retrieval is supposed to have low precision. And a reranker's job is not to improve recall — it cannot, because it only reorders what retrieval already found.
Part 3 — Generation Metrics
Retrieval was fine. Was the answer?
| Metric | Question it answers | How it fails |
|---|---|---|
| Faithfulness / Groundedness | Is every claim supported by the retrieved context? | Detects hallucination — the most important generation metric |
| Answer Relevance | Does the answer address the question asked? | Detects fluent, well-sourced, off-topic answers |
| Answer Correctness | Does it match the ground truth? | Requires reference answers |
| Completeness | Did it include everything relevant that was available? | Detects partial answers that omit retrieved caveats |
| Citation Accuracy | Do the cited sources actually support the cited claims? | Deterministic and cheap — you control the context, so verify markers directly |
Faithfulness and correctness are different, and the gap matters. An answer can be perfectly faithful to retrieved context that is itself outdated — faithful and wrong. It can also be correct while unfaithful, when the model used training knowledge that happened to be right. That second case is a failure even though the user got a good answer, because the mechanism that produced it is unreliable and will produce a wrong answer next time.
What not to use
| Metric | Why it fails for RAG |
|---|---|
| BLEU / ROUGE | Measure n-gram overlap with a reference. A correct answer phrased differently scores badly; a wrong answer reusing the reference's wording scores well. Built for translation and summarization, not grounded QA. |
| Exact match | Only meaningful for short factoid answers |
| Embedding similarity to reference | Better than n-gram overlap, but blind to negation — "you may take ibuprofen" and "you may not take ibuprofen" are near-identical vectors |
Part 4 — LLM-as-Judge
Faithfulness cannot be computed with string matching, so the standard approach is to ask a model.
LLM-as-judge for faithfulness
Claim-level scoring beats asking for a holistic 1–10 rating. It is more consistent, and it tells you which sentence was unsupported rather than just producing a number.
Making judges trustworthy
| Practice | Why |
|---|---|
| Validate against humans first | Score ~50 examples by hand and compare. If judge and humans disagree badly, the judge is measuring noise and every downstream number is fiction. |
| Use binary or few-point scales | "Supported: yes/no" is far more consistent than "rate 1–10" |
| Require the reasoning before the verdict | Forces the judgement to be grounded and gives you something to inspect |
| Fix temperature at 0 | Evaluation must be reproducible |
| Use a strong model | A weak judge produces confident noise; this is the wrong place to economize |
Known biases
| Bias | Effect | Mitigation |
|---|---|---|
| Self-preference | Models favour text from the same family | Judge with a different model than you generate with |
| Position bias | In A/B comparisons, one position wins more | Randomize order; run both orders and average |
| Length bias | Longer answers score higher regardless of quality | Instruct explicitly that length is not a virtue |
| Fluency bias | Well-written wrong answers beat clumsy right ones | Force claim-by-claim checking rather than an overall impression |
The RAG Triad
A compact, widely-used framing — three checks over the same triangle:
The RAG triad
Context Relevance
Groundedness
Answer Relevance
If all three hold, the answer is almost certainly good. When one fails, it names the broken stage: context relevance points at retrieval, groundedness at the prompt, answer relevance at query understanding.
Part 5 — Offline and Online
Offline evaluation runs against your golden dataset. Online evaluation measures reality.
| Signal | What it tells you | Caveat |
|---|---|---|
| Thumbs up/down | Direct quality feedback | Very sparse and skewed toward complaints |
| Citation click-through | Users verifying sources — a trust signal | High rates can mean either engagement or distrust |
| Query reformulation rate | User rephrased immediately → the first answer failed | One of the strongest implicit quality signals |
| Conversation abandonment | User left without resolution | Ambiguous — may mean satisfied |
| Escalation to human support | Unambiguous failure, and easy to price | The best business-aligned metric available |
| Refusal rate | Trending up means retrieval is degrading | Watch it as a leading indicator |
Reformulation rate and escalation rate are the two online metrics worth instrumenting first. Both are automatic, neither depends on users choosing to give feedback, and both map directly onto "the system failed this person."
Evaluation in CI
Treat evaluation like a test suite:
Evaluation-driven development
Change something
New chunk size, different embedding model, new prompt
Run the golden dataset
Full retrieval and generation metrics
Compare against the baseline
Per-metric deltas, not a single aggregate
Ship and watch online signals
Offline gains must show up in reality
Always compare against a stored baseline, never against memory. Version your evaluation runs alongside your code. Without a baseline, gradual regression is invisible — every individual change looks acceptable while the system slowly degrades.
Common Evaluation Mistakes
What goes wrong in evaluation
Measuring only end-to-end quality
Tells you something is broken, never which half. Split retrieval from generation before anything else.
No 'unanswerable' questions in the test set
You have no measurement of refusal behaviour, so you will optimize toward a system that always answers — that is, toward hallucination.
Trusting an unvalidated LLM judge
If the judge does not agree with humans on a sample you checked by hand, every number it produces is just decoration.
A test set built only from LLM-generated questions
Skews toward what your current retrieval already handles well, so scores look good while real users fail. Mix in real logged queries as soon as you have them.
Optimizing one metric
Push precision alone and you starve the context of evidence; push recall alone and you flood it. Track a small set and watch the trade.
Evaluating retrieval and generation with the same k
Wide retrieval is supposed to have low precision. Judging it by post-rerank standards produces the wrong conclusion.
Never re-examining the golden dataset
Corpora change, users change, and stale labels quietly become wrong. Revisit it on a schedule.
Concept Checks
Check yourself
Hit Rate@5 is 1.0 but Recall@5 is 0.2. What is happening, and does it matter?
At least one relevant chunk is in the top 5, but 80% of the relevant evidence is missing. It matters whenever answers require synthesis — a question needing three supporting facts will get one, and the model will answer confidently from partial evidence. Hit Rate is satisfied by a single lucky result and hides exactly this, which is why it should never be your only retrieval metric.
Why measure recall before reranking and precision after?
Because each stage has a different job. Initial retrieval casts wide so the right chunks are present — low precision there is expected and harmless. The reranker then reorders that candidate set for precision. Crucially, a reranker can only reorder what retrieval already returned: if recall@50 missed the answer, no reranking recovers it. Measuring precision at the wide stage would push you to narrow retrieval, which is precisely the wrong move.
An answer scores 1.0 faithfulness but users say it is wrong. How?
Faithfulness measures agreement between answer and retrieved context, not truth. If the retrieved chunk is an outdated policy, an answer that reproduces it perfectly is fully faithful and factually wrong. The defect is upstream — stale documents, missing recency metadata, or a retriever preferring an old version over its replacement. Faithfulness is necessary and not sufficient; correctness needs ground truth.
Why is a test set with no unanswerable questions dangerous?
Because it makes hallucination invisible and refusal costly. Every question having an answer means a model that always answers scores perfectly, while a well-calibrated system that refuses when evidence is absent is never rewarded for it. Every tuning decision then pushes toward answering regardless of evidence. Unanswerable questions are the only way to measure whether "I don't know" works.
Why is query reformulation rate such a strong online signal?
Because it is an unambiguous, automatic judgement made by the user at the moment of failure. Someone who immediately rephrases the same question has told you the first answer did not serve them — no survey, no thumbs, no self-selection. Unlike explicit feedback, which a small and unrepresentative slice of users provide, reformulation is captured for everyone, and unlike abandonment it is hard to read as satisfaction.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Split the evaluation | Retrieval and generation fail differently and are fixed differently |
| Golden dataset | Questions + ground truth + relevant chunk IDs; 50–100 to start |
| Hit Rate@k | Any relevant chunk in the top k — coarse, easily flattering |
| Recall@k | Fraction of relevant chunks found — the pre-rerank metric that matters |
| Precision@k | Fraction of returned chunks that are relevant — the post-rerank metric |
| MRR | Rank of the first relevant result, reciprocated |
| nDCG | Graded relevance, rank-discounted, normalized to 0–1 |
| Faithfulness | Every claim supported by context — the anti-hallucination metric |
| Answer relevance | The answer addresses the actual question |
| LLM-as-judge | Necessary for subjective metrics; validate against humans before trusting |
| RAG triad | Context relevance, groundedness, answer relevance |
| Reformulation rate | The strongest cheap online failure signal |
| Baselines in CI | Compare every change against stored results, never against memory |
Next
You can now measure the system. Next, run it: Production & Operations.
RAG Architectures
Naive, Advanced, and Modular RAG — plus HyDE, RAG-Fusion, Self-RAG, CRAG, Adaptive, Agentic, Graph, and Speculative RAG, and how to choose between them
Production & Operations
Latency budgets, caching, cost control, incremental indexing, index migrations, observability, access control, multi-tenancy, and indirect prompt injection