Retrieval Strategies
Dense, sparse, and hybrid search; BM25 and reciprocal rank fusion; query rewriting, expansion, HyDE, decomposition, and routing
Retrieval Strategies
TL;DR
Vector search alone fails on exact terms like product codes and rare names. Keyword search alone fails on paraphrases. Hybrid search runs both and fuses the rankings, and it is the correct default for production. Separately, transforming the user's query before searching — rewriting, expanding, decomposing — often improves results more than any change to the search engine itself.
| Property | Value |
|---|---|
| Level | Intermediate → Advanced |
| Reading time | ~25 minutes |
| Prerequisites | Vector Databases |
| You will understand | Every retrieval method, how to fuse them, and how to fix queries before searching |
Two Kinds of Search
| Dense (vector) | Sparse (keyword) | |
|---|---|---|
| Matches on | Meaning | Literal words |
| Representation | ~1536 numbers, all non-zero | Word counts across a huge vocabulary, nearly all zero |
| Why the name | Vectors are "dense" — every position has a value | Vectors are "sparse" — almost every position is zero |
| Finds | "blood thinner" → documents about warfarin | "WF-2249" → the document containing exactly that |
| Misses | Codes, IDs, rare proper nouns, negation | Synonyms, paraphrases, anything worded differently |
| Classic algorithm | Cosine similarity over embeddings | BM25 |
BM25 in Plain Language
BM25 (Best Match 25) is the standard keyword ranking algorithm, and it remains extremely strong despite being decades old. It scores a document on three intuitions:
- Term frequency — a document mentioning your search word many times is more relevant…
- …with diminishing returns — the tenth mention adds far less than the second.
- Rare words matter more — a document matching "WF-2249" is far more informative than one matching "the". This is IDF (Inverse Document Frequency).
Plus a length correction, so long documents do not win merely by containing more words.
BM25 is not a legacy fallback. On queries containing exact identifiers it outperforms modern embeddings decisively, because embedding models have no meaningful representation for a string they never saw during training. Any serious RAG system runs both.
See the Failure Modes
Three queries, each designed to break one method. Watch what fusion recovers.
Keyword vs vector vs fusion
Three queries, each designed to break one of the two search methods. Compare the top results in each column.
Keyword
BM25 — matches literal words
Protocol WF-2249 defines anticoagulation monitoring for inpatients.
score 9.4
Warfarin interacts with NSAIDs, raising the risk of bleeding.
no match
Blood thinners require regular monitoring of clotting time.
no match
Ibuprofen is a non-steroidal anti-inflammatory drug used for pain.
no match
Vector
Cosine — matches meaning
Protocol WF-2249 defines anticoagulation monitoring for inpatients.
score 0.42
Warfarin interacts with NSAIDs, raising the risk of bleeding.
score 0.39
Blood thinners require regular monitoring of clotting time.
score 0.37
Patients on anticoagulants should report unusual bruising promptly.
score 0.35
Fusion
RRF — combines both ranks
Protocol WF-2249 defines anticoagulation monitoring for inpatients.
rrf 0.0328
Warfarin interacts with NSAIDs, raising the risk of bleeding.
rrf 0.0323
Blood thinners require regular monitoring of clotting time.
rrf 0.0317
Ibuprofen is a non-steroidal anti-inflammatory drug used for pain.
rrf 0.0310
score = 1/(k + keyword_rank) + 1/(k + vector_rank). A small k lets a single first-place finish dominate the fused ranking; a large k flattens the contributions so a document must do well in both lists. 60 is the common default.
Keyword search nails this instantly — the code is a literal string match. Vector search is nearly blind: to an embedding model, "WF-2249" is meaningless noise, so every document looks equally unrelated.
The middle scenario is the one that catches teams out. The best answer never uses the user's words at all — it says "NSAIDs" and "warfarin" where the user said "painkillers" and "blood thinners". Keyword search ranks it third; vector search puts it first. If you have only keyword search, that answer is effectively invisible.
Hybrid Search
Run both searches and combine the rankings.
Reciprocal Rank Fusion
RRF is the standard fusion method, and its strength is that it combines ranks, not scores — which sidesteps the fact that a BM25 score of 9.4 and a cosine score of 0.88 are not remotely comparable quantities.
RRF_score(doc) = Σ 1 / (k + rank_in_list)
lists
k is a constant, conventionally 60.A document ranked 1st by keyword and 3rd by vector, with k=60, scores 1/61 + 1/63 = 0.0323. A document ranked 1st by both scores 1/61 + 1/61 = 0.0328 and wins. Documents strong in both lists rise; documents strong in only one still place respectably.
| k value | Effect |
|---|---|
| Small (1–10) | Rank 1 dominates heavily — a single first place can win outright |
| 60 (default) | Balanced; contributions decay gently |
| Large (100+) | Flattens differences; a document must do well across both lists |
Alternative: Weighted Score Fusion
Normalise both score sets to 0–1 and take a weighted sum:
final = α × normalised_vector_score + (1 − α) × normalised_keyword_score| RRF | Weighted fusion | |
|---|---|---|
| Needs comparable scores | No | Yes — normalisation required |
| Tuning | Just k | α, plus the normalisation method |
| Robustness | High — ranks are stable | Lower — sensitive to score distribution shifts |
| Can weight one source | Not directly | Yes, easily |
| Recommendation | Default | When you specifically need to favour one method |
Start with RRF at k=60. It has one parameter, needs no calibration, and is hard to get badly wrong. Move to weighted fusion only when you have a measured reason to bias toward one retrieval method.
Fixing the Query Before You Search
The largest retrieval gains often come not from better search, but from searching for something better. Users do not write good search queries; they write questions.
Query Rewriting
Resolve pronouns and implicit references against the conversation.
History: "Can I take ibuprofen with warfarin?"
Follow-up: "What about for elderly patients?"
Searching that follow-up directly retrieves generic content about elderly
patients — nothing about the drug interaction.
Rewritten: "ibuprofen warfarin interaction in elderly patients"In any conversational interface, this is mandatory, not optional. Without it, retrieval quality falls off a cliff at turn two and never recovers, because follow-up questions contain almost no standalone searchable content.
Query Expansion
Generate variations and search all of them, then merge results.
Original: "warfarin side effects"
Expanded:
- "warfarin adverse effects"
- "warfarin bleeding risk complications"
- "anticoagulant side effects monitoring"Costs extra searches (cheap and parallelisable) and one small model call. Reliably improves recall, especially when user vocabulary differs from document vocabulary.
HyDE — Hypothetical Document Embeddings
A clever inversion. Instead of embedding the question, ask a model to write a hypothetical answer, then embed that and search with it.
HyDE
The insight: questions and answers are different kinds of text. A question is short and interrogative; a document passage is declarative and detailed. Searching with answer-shaped text matches answer-shaped documents more effectively.
| Pros | Cons |
|---|---|
| Strong on vague or under-specified queries | Adds a model call — latency and cost |
| Bridges vocabulary gaps well | Can drift off-topic on niche subjects |
| Needs no training | Unhelpful when the query is already precise |
Built fully in the HyDE RAG project.
Query Decomposition
Split multi-part questions into independent retrievals.
"What is the maximum dose of warfarin and what should I do if I miss one?"
→ "warfarin maximum dose"
→ "warfarin missed dose instructions"One search for a compound question retrieves a compromise between two topics and often serves neither well.
Step-Back Prompting
Ask a broader question first to retrieve grounding context, then the specific one.
Specific: "Can a 78-year-old on warfarin take ibuprofen for arthritis?"
Step-back: "How do NSAIDs interact with anticoagulants?"The general question retrieves the principle; the specific one retrieves the detail. Together they give the model both.
Query Routing
Not every question needs the same treatment — or any retrieval at all.
Route before you retrieve
No retrieval needed
"Thanks", "can you rephrase that?", "hello". Retrieval adds latency, cost, and irrelevant context. Answer directly.
Single-source retrieval
A focused factual question. Standard hybrid search over one collection.
Multi-source retrieval
"Compare our 2024 and 2025 policies." Retrieve from both, keep them separate so the model can contrast them.
Tool call, not retrieval
"What is my current order status?" That is a live database lookup. RAG over documents cannot answer it, and will confidently produce something wrong if you let it try.
Routing is orchestration — see the LLM Router project and the Adaptive RAG project.
Advanced Retrieval Patterns
| Pattern | What it does | Use when |
|---|---|---|
| MMR (Maximal Marginal Relevance) | Balances relevance against diversity, so results are not five near-copies | Overlapping chunks crowd your results |
| Multi-vector | Store several vectors per chunk (summary, questions it answers, raw text) | Chunks are matched for different reasons by different users |
| ColBERT / late interaction | Store per-token vectors and match at token level | Maximum precision is worth the storage |
| Self-query | Model extracts metadata filters from the natural-language query | Users naturally say "policies from last year" |
| Sentence-window | Match on a single sentence, return surrounding sentences | You want pinpoint matching plus context |
| Iterative / multi-hop | Retrieve, read, then retrieve again with what you learned | Questions requiring chained facts |
On MMR
MMR deserves attention because chunk overlap makes near-duplicate results common. Retrieving five chunks that are 90% the same text wastes four slots. MMR explicitly penalises a candidate for resembling what you already selected.
How Many to Retrieve
| Setting | Typical value | Reasoning |
|---|---|---|
| Candidates from search | 20–100 | Cheap. Favour recall — you cannot rerank what you did not retrieve |
| After reranking | 3–8 | Precision. This is what the model actually reads |
| Token budget for context | 2,000–8,000 | Depends on model, cost tolerance, and answer complexity |
More context is not better past a point. Extra chunks dilute the signal, cost tokens on every request, and increase the chance the model latches onto something tangential. If you are retrieving 20 chunks into the final prompt, you almost certainly need a reranker rather than a bigger context.
Choosing Your Setup
| Situation | Recommended retrieval |
|---|---|
| Prototype, small corpus | Dense only — simplest thing that works |
| Any production system | Hybrid (dense + BM25) with RRF |
| Corpus full of codes, SKUs, IDs | Hybrid, weighted toward keyword |
| Conversational interface | Hybrid plus mandatory query rewriting |
| Vague or exploratory queries | Hybrid plus HyDE or expansion |
| Compound questions | Hybrid plus decomposition |
| Mixed query types | Add routing in front |
Concept Checks
Check yourself
Why does RRF combine ranks instead of scores?
Because the scores are incommensurable. BM25 produces unbounded positive numbers whose scale depends on corpus statistics; cosine similarity produces values in a narrow band near the top of −1 to 1. Adding or comparing them directly is meaningless, and normalising them introduces its own sensitivity to distribution shifts. Ranks are ordinal, stable, and comparable by construction.
Users search by internal project codenames and retrieval fails. Diagnosis?
You are almost certainly running dense-only search. Codenames are rare tokens the embedding model has essentially no representation for, so every document looks equally unrelated. Add BM25 and fuse — keyword search treats a rare exact string as a very strong signal, which is precisely the right behaviour here.
Why can a factually wrong HyDE draft still improve retrieval?
Because the draft is never shown to anyone and never contributes to the answer — it is used only as a search vector and then discarded. What matters is that it has the shape and vocabulary of a real document rather than a question, so it matches document embeddings more closely. Retrieval then returns real documents, which supply the actual facts.
You retrieve 25 chunks straight into the prompt and quality is poor. What is the fix?
Add a reranker and cut to 3–8. Twenty-five chunks means most of the context is irrelevant to the specific question, which dilutes the signal, raises cost on every call, and gives the model more opportunities to anchor on something tangential. Retrieve wide by all means — then narrow before generating.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Dense retrieval | Vector search on meaning; misses exact identifiers |
| Sparse retrieval | BM25 on literal words; misses paraphrases |
| Hybrid | Both, fused — the production default |
| RRF | Fuses by rank, not score; k=60 is the standard |
| Query rewriting | Resolves pronouns; mandatory in conversation |
| Query expansion | Multiple phrasings to bridge vocabulary gaps |
| HyDE | Search with a hypothetical answer, not the question |
| Decomposition | Split compound questions into separate retrievals |
| Routing | Decide whether and where to retrieve |
| MMR | Trades a little relevance for diversity |
| Retrieve wide, use narrow | 20–100 candidates → 3–8 final chunks |
Next
You have candidates. Now pick the best few and assemble them well: Reranking & Context Assembly.
Vector Databases & Indexes
How approximate nearest neighbour search works — HNSW, IVF, quantization — plus metadata filtering, and how to choose a database
Reranking & Context Assembly
Cross-encoders, ColBERT, and LLM rerankers — plus context ordering, the lost-in-the-middle effect, deduplication, and compression