Failure Modes & Debugging
A systematic method for diagnosing bad answers — symptom to root cause to fix, across parsing, chunking, embedding, retrieval, reranking, and generation
Failure Modes & Debugging
TL;DR
When a RAG system gives a bad answer, the instinct is to edit the prompt. That is usually the wrong end of the pipeline. This page gives you a bisection method — one question that splits the problem in half — followed by a symptom-to-cause catalogue for every stage. Diagnose before you fix.
| Property | Value |
|---|---|
| Level | Intermediate → Advanced |
| Reading time | ~25 minutes |
| Prerequisites | Production & Operations |
| You will understand | How to locate any RAG bug systematically instead of by guesswork |
The One Question That Splits Every Bug
Was the correct text in the context window?
Answer that before touching anything. It divides every possible cause into two separate halves, and the two halves share no fixes at all.
The bisection
The right chunk was NOT there
A retrieval-side bug: parsing, chunking, embedding, indexing, search, or filters. Prompt changes cannot help.
The right chunk WAS there
A generation-side bug: context assembly, ordering, prompt, or model behaviour. Retrieval tuning cannot help.
If you can only take one thing from this page: log retrieved chunk IDs, and read them first.
Work a real bug through the tree:
Find the cause of a bad answer
Was the correct text actually in the context window?
Look at the logged chunk IDs for the failing query. This one question splits every RAG bug in half, and the two halves share no fixes. Never skip it.
Every path starts with the same question, and that is deliberate. Once you know whether the right text reached the model, half the possible causes disappear. Debugging a prompt when the real bug is in your chunking is the most common wasted week in RAG work.
Then bisect again
Once you know the side, narrow further:
Retrieval-side bisection
Each step is a cheap, concrete check. Working through them in order beats any amount of intuition.
Part 1 — Retrieval-Side Failures
Ingestion and parsing
| Symptom | Likely cause | Fix |
|---|---|---|
| Answers about a whole document class are always wrong | That format never parsed — scanned PDFs, complex tables | Check parser output per format; add OCR or a layout-aware parser |
| Table figures are wrong or nonsensical | Table flattened into unordered text, losing row/column association | Extract tables structurally; serialize each row as a self-contained sentence |
| Text from multi-column pages is interleaved gibberish | Naive extraction read across columns instead of down them | Use a layout-aware parser that recovers reading order |
| Recently added documents are never retrieved | Indexing job failed, or the document was skipped silently | Alert on indexing failures; reconcile source count against index count |
Read your parsed text. Not a sample of five — a real sample across every format you ingest. Parsing failures are invisible from the outside because the system keeps working; it simply answers from documents whose content it mangled. This is the most under-diagnosed stage in RAG.
Chunking
| Symptom | Likely cause | Fix |
|---|---|---|
| Answers are half-right, missing a condition or exception | The qualifying sentence landed in the adjacent chunk | Increase overlap; chunk on semantic boundaries |
| "According to the table above…" with no table | Chunk boundary separated prose from its referent | Keep structural elements with their context; parent-document retrieval |
| Retrieved chunks are topically right but never specific | Chunks too large — one embedding averaging several topics into mush | Reduce chunk size; split on headings |
| Retrieved chunks lack the context to be interpretable | Chunks too small; pronouns and references dangle | Increase size, or prepend section headings to each chunk |
| A fact stated once is never found | It sits in a sentence whose subject is only named in the previous paragraph | Contextual chunk enrichment — prepend a summary of the parent section |
Embedding and indexing
| Symptom | Likely cause | Fix |
|---|---|---|
| Results are near-random for every query | Query and documents embedded with different models, or a mixed index after a partial migration | Verify one model everywhere; rebuild the index |
| Asymmetric search is poor | Model needs query/document prefixes and they were omitted | Apply the prefixes the model documents |
| Long chunks retrieve badly | Chunks exceed the embedding model's token limit and were silently truncated | Check token counts against the model limit before embedding |
| Recall dropped after an infra change | ANN index parameters changed — efSearch, nprobe lowered | Restore parameters; measure the recall/latency trade explicitly |
| Non-English content retrieves poorly | Monolingual embedding model | Use a multilingual model |
Silent truncation is a classic. Many embedding APIs accept over-long input and quietly embed only the first N tokens. Everything appears to work; the tail of every long chunk simply does not exist as far as search is concerned. Count tokens explicitly at ingestion.
Search and ranking
| Symptom | Likely cause | Fix |
|---|---|---|
| Exact identifiers — SKUs, error codes, names — are never found | Pure dense search; embeddings are poor at rare exact tokens | Hybrid search. This is the textbook case for BM25 |
| Paraphrased questions fail | Pure keyword search | Hybrid search, from the other direction |
| Relevant results sit just outside top-k | k too small, or no reranking | Retrieve wider, then rerank down |
| Results ignore an explicit constraint like a year or region | Constraint expressed only in the text, not applied as a filter | Extract constraints into metadata filters |
| Filtered searches return too few results | Post-filtering applied after ANN search discarded most candidates | Use pre-filtering, or over-retrieve before filtering |
| Answers come from an outdated document | Both versions indexed; no recency signal | Delete superseded chunks; add recency metadata and boost |
| Follow-up questions in a conversation fail | Query sent verbatim — "what about the second one?" is unsearchable | Query rewriting to a standalone question |
Part 2 — Generation-Side Failures
The correct chunk was in the context. The answer is still wrong.
| Symptom | Likely cause | Fix |
|---|---|---|
| Model states facts not in the context | No explicit grounding instruction; blending training knowledge | "Answer using only the context"; require per-claim citations |
| Model invents rather than admitting a gap | No acceptable refusal path | Give exact refusal wording and one example |
| Model uses only the first and last chunks | Lost in the middle — attention degrades over long contexts | Fewer chunks; put the strongest first; rerank harder |
| Answer contradicts a retrieved chunk | Two chunks disagree; the model silently picked one | Instruct it to surface conflicts; use recency metadata |
| Citations point at the wrong source | Model misattributed | Verify markers programmatically against the real chunk list |
| Citations reference sources that do not exist | Context lacked attribution markers, so references were invented | Number every chunk in the context block |
| Answers vary run to run | Temperature too high | Temperature 0–0.3 |
| Answers are fluent but do not address the question | Question buried before a large context | Question last; shrink the context |
| Model follows instructions found inside a document | Indirect prompt injection | Delimit context as data; control corpus write access; no privileged tools on this path |
Lost in the middle is worth remembering because it feels backwards: adding more retrieved context can make answers worse. Beyond a certain length, material in the middle of the context is attended to less reliably than material at either end. Four well-ranked chunks routinely outperform twenty mediocre ones.
Part 3 — Systemic Failures
These do not present as one bad answer. They present as a system that is quietly worse than you think.
Failures that hide
Confident answers when retrieval returned nothing
The empty-retrieval path falls through to ungrounded generation. Every dashboard looks healthy while the system produces exactly the hallucinations RAG exists to prevent. Make empty retrieval an explicit, logged, refusing branch.
Gradual quality decay with no incident
The corpus grew, the query distribution shifted, documents went stale. Nothing broke; everything degraded. Only continuous evaluation against a stored baseline detects this.
Works for you, fails for users
You test with well-formed questions using document vocabulary. Real users write fragments, typos, and their own words. Build the golden dataset from real logs.
A wrong answer served from cache
Indistinguishable from a model error to the user and much harder to trace. Log every cache hit with the layer and the key that matched.
Cross-tenant answers
Cache keys omitting tenant identity, or missing retrieval-time ACL filters. A correctness bug and a data breach at once.
Evaluation scores improve, users do not benefit
The test set drifted from reality — often because it was LLM-generated from chunks your retrieval already handles well. Refresh it from production queries.
The Debugging Checklist
Work top to bottom. Do not skip ahead.
| # | Check | Rules out |
|---|---|---|
| 1 | Is the fact present in the source documents? | Ingestion gaps |
| 2 | Did the parser extract it readably? | Parsing failures |
| 3 | Is it intact inside one chunk? | Boundary damage |
| 4 | Does that chunk exist in the index? | Indexing failures |
| 5 | Does it rank for the query? | Ranking and model issues |
| 6 | Did a filter remove it? | Metadata and permission filters |
| 7 | Did it survive reranking into the final context? | Reranker misbehaviour |
| 8 | Was it positioned where the model would attend to it? | Lost in the middle |
| 9 | Did the prompt constrain the model to use it? | Grounding failures |
| 10 | Was the answer served from cache? | Stale or near-miss cache hits |
Steps 1–4 are pure data checks that need no model at all, and they catch a large share of real bugs. They are also the steps most often skipped, because editing a prompt feels more like progress than printing chunks.
Concept Checks
Check yourself
A user reports a wrong answer. What is your first action?
Pull the logged chunk IDs for that query and check whether the supporting chunk was in the context. That single fact splits the space of causes in half and determines which half of the pipeline to investigate. Any other first action — reading the prompt, adjusting temperature, trying the query yourself — risks a long investigation in the wrong half.
Product codes are never retrieved, but conceptual questions work well. Why?
Dense embeddings represent meaning, and a code like XR-4471B carries almost no meaning the model ever learned. It sits nowhere useful in vector space, and may not even survive tokenization as a single unit. Keyword search handles it trivially, since it matches the literal string. This is the textbook argument for hybrid search: the two methods fail on opposite inputs.
Answers are consistently half-right — correct rule, missing exception. Where do you look?
Chunking. The pattern of a right main statement plus a missing qualifier is the signature of a boundary landing between a rule and its exception, so the retrieved chunk genuinely contains only half the fact. Print the chunks around that content and look at where the splits fall. No prompt or retrieval change recovers text that was never in the same chunk.
You added more retrieved chunks and quality got worse. What happened?
Two effects compound. Extra chunks rank lower, so they are mostly noise, and they water down the signal. Longer contexts also suffer lost-in-the-middle: material away from the edges gets read less carefully. That can include the good chunks you already had. More context is not more information. Retrieve wide, rerank hard, and pass few high-quality chunks.
Retrieval works in testing but fails for one customer in production. What do you suspect?
Filters. A tenant or permission filter is excluding candidates before scoring, either because ACL metadata on those chunks is missing or stale, or because post-filtering ran after ANN search had already discarded the relevant candidates. Both are invisible in single-tenant testing. Log the filter applied alongside the pre-filter and post-filter candidate counts.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| The bisection | Was the correct chunk in the context? Answer first, fix second |
| Log chunk IDs | The prerequisite for all systematic debugging |
| Parsing failures hide | Read your parsed text across every format |
| Half-right answers | Almost always chunk boundaries |
| Silent truncation | Over-long chunks embedded only in part |
| Exact terms fail on dense search | The textbook hybrid-search case |
| Post-filtering starves results | Filters applied after ANN search return too little |
| Lost in the middle | More context can mean worse answers |
| Stale duplicates | Superseded chunks compete and sometimes win |
| Empty retrieval must refuse | Otherwise failure is invisible and ungrounded |
| The 10-step checklist | Work it in order; steps 1–4 need no model |
Next
Time to put every piece together on one realistic brief: Designing a RAG System.
Production & Operations
Latency budgets, caching, cost control, incremental indexing, index migrations, observability, access control, multi-tenancy, and indirect prompt injection
Designing a RAG System
A complete worked example — gathering requirements, sizing the index, choosing every component with reasons, and planning the rollout