RAG Architectures
Naive, Advanced, and Modular RAG — plus HyDE, RAG-Fusion, Self-RAG, CRAG, Adaptive, Agentic, Graph, and Speculative RAG, and how to choose between them
RAG Architectures
TL;DR
Everything so far described one shape: retrieve once, then generate once. That shape is called Naive RAG, and it is the right starting point for every project. Each named architecture on this page repairs one specific failure of that baseline: a query the embedding cannot match, a retrieval that came back wrong, a question that needs two hops. Learn them as fixes with a cost, not as an upgrade ladder you are supposed to climb.
| Property | Value |
|---|---|
| Level | Intermediate → Advanced |
| Reading time | ~30 minutes |
| Prerequisites | Generation & Grounding |
| You will understand | Every major RAG architecture, the failure each one repairs, and what it costs |
The Three Generations
The literature groups RAG systems into three broad generations. They are a useful map, but they describe increasing modularity, not increasing quality.
The three generations
Naive RAG
Advanced RAG
Modular RAG
| Generation | What defines it | Control flow |
|---|---|---|
| Naive RAG | Retrieve → generate, one pass | Fixed, linear |
| Advanced RAG | Same shape, with optimizations bolted before and after retrieval | Fixed, linear |
| Modular RAG | Retrieval becomes a component that can be skipped, repeated, routed, or replaced | Dynamic, branching |
The generations are not a maturity ladder. A well-tuned Naive RAG with good chunking, hybrid search, and a reranker beats a badly-tuned Agentic RAG on almost every axis — accuracy included, and by an enormous margin on latency and cost. Move up only when you have measured a failure the current architecture cannot fix.
Naive RAG — the baseline
Naive RAG
| Aspect | Detail |
|---|---|
| Cost | 1 embedding call + 1 search + 1 LLM call |
| Latency | Typically 1–3 seconds end to end |
| Strength | Simple, cheap, debuggable, predictable |
| Fails when | Query wording differs from document wording; the answer needs several documents combined; retrieval returns nothing relevant and nobody notices |
This is where every project should start. Build it, measure it, and let the measurements tell you which of the following you actually need.
Part 1 — Advanced RAG
Same linear shape; extra work before and after the search.
Pre-retrieval: fixing the query
HyDE — Hypothetical Document Embeddings
The problem it fixes: a question and its answer are written in different registers. "Why is my laptop so slow?" shares almost no vocabulary with a technical document titled "Thermal throttling under sustained load." Embedding the question searches for text that looks like a question, when you want text that looks like an answer.
HyDE
The hypothetical document being factually wrong is fine. It is never shown to the user and never used as a source — it exists only to produce a better search vector. Its job is to be shaped like the answer, not to be the answer.
| Trade-off | Detail |
|---|---|
| Gain | Large improvement on vague, conversational, or jargon-mismatched queries |
| Cost | One extra LLM call before every search — adds latency to every query |
| Risk | On queries about rare entities the model knows nothing about, the fake answer drifts off-topic and hurts retrieval |
Query rewriting, expansion, and decomposition
| Technique | What it does | Best for |
|---|---|---|
| Rewriting | Turns a messy or context-dependent question into a clean standalone one | Conversational systems where "what about the second one?" must become a full question |
| Expansion | Generates several phrasings and searches with all of them | Recall problems from vocabulary mismatch |
| Decomposition | Splits a multi-part question into sub-questions searched separately | "Compare X and Y" — one search cannot serve both halves |
RAG-Fusion
Query expansion plus Reciprocal Rank Fusion (RRF): generate several rewrites, search with each, then merge the ranked lists by summing 1 / (k + rank) for every document across every list.
RAG-Fusion
Rewrite 1
Search → ranked list A
Rewrite 2
Search → ranked list B
Rewrite 3
Search → ranked list C
Documents that rank well across multiple phrasings rise to the top. It is a consensus mechanism: agreement across independent searches is evidence of genuine relevance.
Post-retrieval: fixing the results
Covered in depth in Reranking & Context Assembly — reranking, deduplication, compression, and ordering around the lost-in-the-middle effect. These are the highest value-per-unit-complexity additions in all of RAG.
The pragmatic stack
Hybrid search + a cross-encoder reranker + good chunking. That combination is still linear, still cheap, still easy to debug, and resolves the large majority of retrieval failures. Exhaust it before reaching for anything below.
Part 2 — Modular RAG
Here the control flow stops being a straight line. Retrieval becomes something the system decides about.
Self-RAG — the model critiques itself
The model emits reflection tokens — special markers judging its own process: is retrieval needed here?, is this passage relevant?, is my sentence supported by it?
Self-RAG
Assess the question
Decide whether retrieval is needed at all
Retrieve (if needed)
Skipped entirely for questions answerable without sources
Grade each passage
Relevant / irrelevant, per passage
Generate
Write the answer from surviving passages
Critique the output
Is every claim supported? Is the question answered?
The genuinely interesting idea is the first step: "Hello" and "what is 2+2" need no retrieval, and a system that retrieves unconditionally wastes money and pollutes its own context on every such turn.
| Trade-off | Detail |
|---|---|
| Gain | Fewer unsupported claims; skips pointless retrievals |
| Cost | Several LLM calls per query; the full version needs a specially fine-tuned model |
| Practical note | The self-critique pattern can be approximated with ordinary prompting on any model — the reflection-token training is not required to get most of the benefit |
Corrective RAG (CRAG) — a fallback when retrieval fails
A lightweight evaluator grades retrieved documents before generation, and the grade selects a strategy.
| Grade | Meaning | Action |
|---|---|---|
| Correct | Documents clearly answer the question | Refine and use them |
| Ambiguous | Partially relevant | Combine internal documents with an external search |
| Incorrect | Nothing relevant found | Discard everything; fall back to web search |
CRAG's usual fallback is web search, which quietly changes your system's security and correctness posture: answers can now come from outside your controlled corpus. For an internal policy assistant that is often unacceptable. The valuable, domain-independent half of CRAG is the evaluator — knowing retrieval failed is what lets you refuse honestly instead of answering from noise.
Adaptive RAG — route by question complexity
One classifier at the front sends each query down the cheapest sufficient path.
Adaptive RAG
Simple
No retrieval — answer directly. "Hello", "what is an API?"
Single-step
Standard retrieve → generate. Most questions land here.
Multi-step
Iterative retrieval with a reasoning loop
This is the best cost-to-benefit ratio of any modular pattern. Real traffic is dominated by simple questions, and routing them away from the expensive path cuts average cost and latency substantially while improving quality on the hard tail that now gets the expensive treatment.
Agentic RAG — retrieval as a tool
The retriever stops being a pipeline stage and becomes a tool an agent may call, on its own initiative, as many times as it decides.
Agentic RAG
Plan
What do I need to know to answer this?
Choose a tool
Vector search, keyword search, SQL, an API, a calculator
Observe results
Read what came back
Synthesize
Combine everything gathered across iterations
| Aspect | Detail |
|---|---|
| Genuinely enables | Multi-hop questions, comparisons across sources, mixing structured and unstructured data |
| Cost | 5–20× a naive query; latency in tens of seconds |
| Hard parts | Loop termination, cost ceilings, unpredictable runs, and debugging a different execution path on every run |
Agentic RAG is the most over-adopted pattern in the field. It is genuinely necessary for open-ended research questions spanning many different kinds of source. It is genuinely wasteful for "what is our refund window?" — which is what most users actually ask. Route into it (see Adaptive RAG); do not make it the default path.
Graph RAG — retrieval over a knowledge graph
Documents are converted into entities (nodes) and relationships (edges). Retrieval traverses that structure instead of only ranking passages.
| Aspect | Detail |
|---|---|
| Solves | Questions no chunk contains the answer to — "which of our suppliers are indirectly exposed to this sanctioned entity?" — where the answer lives in the connections between documents |
| Also solves | Global questions like "what are the main themes across these 500 reports?", via community summarization |
| Cost | Expensive extraction pass over the whole corpus; a graph store to maintain; fragile re-indexing when documents change |
| Use when | Relationships between entities are the actual subject matter — compliance, fraud, literature reviews, org knowledge |
Speculative RAG — parallel drafts
Generate several answers concurrently from different retrieved subsets using a small fast model, then have a larger model verify and pick the best.
Speculative RAG
Small model → draft A
From subset 1
Small model → draft B
From subset 2
Small model → draft C
From subset 3
Trades money for wall-clock latency and robustness: the drafts run in parallel, so the user waits for one draft plus one verification rather than for a long sequential chain.
Long-context RAG — fewer, bigger chunks
With very large context windows, one strategy is to retrieve far less aggressively and pass whole documents.
Long context does not delete RAG. It changes the ratio — you retrieve fewer, larger units instead of many small ones. You still cannot fit a corpus in a window. You still pay per token. Attention still fades in the middle of very long inputs. And precision still falls when you bury the relevant paragraph in 200 pages of noise. Retrieval remains the mechanism that decides what the model looks at.
Choosing an Architecture
Match the symptom to the architecture
You are starting a new project
RecommendedNaive RAG, done carefully. Good parsing, sensible chunking, hybrid search, a reranker. Measure before adding anything.
Users phrase questions unlike your documents
HyDE or query rewriting. Cheapest first: try rewriting before HyDE, since it adds less latency risk.
Recall is poor and results are inconsistent across phrasings
RAG-Fusion. Multiple rewrites plus reciprocal rank fusion surfaces documents that survive several phrasings.
The system confidently answers when retrieval found nothing
CRAG's evaluator. Grade retrieval quality and refuse honestly on a failed grade — with or without the web-search fallback.
Cost and latency are dominated by simple questions
Adaptive RAG. Route simple queries away from the expensive path. Usually the highest-ROI modular pattern.
Questions need several linked lookups to answer
Agentic RAG or query decomposition. Try decomposition first — it is far cheaper and deterministic.
The answers live in relationships between entities
Graph RAG. The only pattern here that changes what is representable, not just what is retrievable.
You adopted an architecture because it was new
The most common and most expensive mistake. Every pattern here multiplies cost, latency, and debugging difficulty. Each one must be justified by a measured failure of the simpler system.
Cost and latency at a glance
Run the same question through each architecture and watch the clock and the bill:
What each architecture actually costs
Retrieve once, generate once. The baseline every project should start from.
Turn the question into a vector
Find the 5 closest chunks
Write the answer from those chunks
Time so far
0 ms
Baseline was 965 ms
Cost so far
0.00 units
Baseline was 1.01
Run all five. Two results surprise most people: reranking makes queries cheaper, not dearer, because it shrinks the context you pay for. And Adaptive RAG beats the baseline on both axes, because it stops you spending a full retrieval and generation on questions that never needed one.
Rough relative figures — measure your own, but the ordering is stable.
| Architecture | LLM calls/query | Relative latency | Relative cost |
|---|---|---|---|
| Naive RAG | 1 | 1× | 1× |
| + Reranking | 1 (+1 small model) | 1.2× | 1.1× |
| HyDE | 2 | 1.8× | 1.8× |
| RAG-Fusion | 2–4 | 2× | 2.5× |
| Adaptive RAG | 1–4 | 0.7× average | 0.6× average |
| Self-RAG | 3–6 | 3× | 4× |
| CRAG | 2–4 | 2.5× | 3× |
| Speculative RAG | 4–6 (parallel) | 1.5× | 4× |
| Agentic RAG | 5–20 | 10× | 15× |
Adaptive RAG is the only row that can come in below the baseline, because it removes retrieval and generation work from the many queries that never needed it. That is why it is usually the first modular pattern worth adopting.
Combining Architectures
These are not mutually exclusive, and production systems layer them:
A realistic production composition
Every stage there is independently measurable and independently removable — which is exactly the property you want when something regresses at 3am.
Concept Checks
Check yourself
Why does HyDE work even when the hypothetical document is factually wrong?
Because it is used only as a search vector, never as a source. Retrieval matches text by shape and vocabulary, and an invented answer occupies the same region of embedding space as a real answer on that topic — technical register, domain nouns, declarative phrasing. Its factual content is discarded; only its geometry is used. The real retrieved chunks supply every fact in the final answer.
Adaptive RAG adds a classifier call to every query. Why can it still be cheaper than the baseline?
Because the classifier is small and cheap, while the work it avoids is large and expensive. Real traffic contains many greetings, clarifications, and questions needing no corpus lookup at all. You pay a tiny routing cost on every query, and in exchange you skip embedding, search, and a large stuffed-context generation on a good fraction of them. That trade comes out ahead. The hard queries that remain also get a bigger budget than a one-size pipeline would have given them.
When is Graph RAG genuinely necessary rather than a more elaborate way to do the same thing?
When the answer exists only in relationships spanning documents, so no chunk contains it. "Which suppliers connect to this sanctioned entity through two intermediaries?" cannot be retrieved by similarity at any chunk size, because no passage states that fact — it emerges from traversing edges. Contrast with "what is our refund window?", where a single chunk holds the answer and a graph adds cost with no new capability.
Your Agentic RAG scores highest on accuracy in evaluation. Why might it still be the wrong choice?
Because accuracy is one axis among several. It costs roughly 15× and takes roughly 10× longer. It runs a different path every time, which makes failures hard to reproduce. And it may have won its margin on a handful of hard queries while being pure waste on the rest. The right move is usually to route: send the hard tail to the agent and everything else to the cheap path, capturing the accuracy gain without paying for it on every query.
Long-context models keep growing. Does that make retrieval obsolete?
No — it moves the boundary without removing it. Corpora are larger than any window, tokens are billed on every call, attention degrades over very long inputs, and precision drops when the relevant paragraph sits inside hundreds of pages of irrelevant text. Bigger windows let you retrieve fewer and larger units — whole documents instead of paragraphs — which is a change in granularity, not the elimination of the decision about what the model reads.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Naive RAG | Retrieve once, generate once — the correct starting point |
| Advanced RAG | Same linear shape plus pre- and post-retrieval optimizations |
| Modular RAG | Retrieval becomes conditional, repeatable, and routable |
| HyDE | Search with an invented answer instead of the question |
| RAG-Fusion | Several rewrites, searched separately, fused with RRF |
| Self-RAG | The model grades its own retrieval and output, and may skip retrieval |
| CRAG | Grade retrieval quality; fall back when it failed |
| Adaptive RAG | Route by complexity — the cheapest sufficient path per query |
| Agentic RAG | Retrieval as a tool an agent calls repeatedly; powerful and expensive |
| Graph RAG | Traverse entities and relationships; answers that live between documents |
| Speculative RAG | Parallel drafts from a small model, verified by a large one |
| Long-context RAG | Fewer, larger retrieved units — not the end of retrieval |
| The governing rule | Every architecture is a fix for a measured failure, never an upgrade for its own sake |
Next
You have the architectures. Now learn how to tell whether any of them actually helped: Evaluation & Metrics.
Multimodal RAG
Retrieving from images, tables, charts, and scanned pages — captioning, multimodal embeddings, vision models, and picking between them
Evaluation & Metrics
Measuring RAG properly — recall@k, precision@k, MRR, nDCG, faithfulness, answer relevance, golden datasets, LLM-as-judge, and online signals