Reranking & Context Assembly
Cross-encoders, ColBERT, and LLM rerankers — plus context ordering, the lost-in-the-middle effect, deduplication, and compression
Reranking & Context Assembly
TL;DR
Search is fast but approximate at judging relevance, because it compared the query and each document separately. A reranker reads them together and scores their actual relationship — far more accurately, and far too slowly to run over a whole corpus. So you retrieve wide, rerank narrow. Then how you order and assemble the survivors materially changes the answer.
| Property | Value |
|---|---|
| Level | Intermediate → Advanced |
| Reading time | ~20 minutes |
| Prerequisites | Retrieval Strategies |
| You will understand | Reranker types, when they pay for themselves, and how to build a good context block |
Why Search Alone Misjudges Relevance
Vector search uses a bi-encoder: the document was embedded weeks ago, the query is embedded now, and the two vectors are compared. Neither embedding ever saw the other text.
That is what makes it fast — document vectors are pre-computed and indexed. It is also what makes it imprecise: the model had to compress each document into a fixed vector without knowing what would be asked.
Bi-encoder vs cross-encoder
Bi-encoder — used for search
Embeds query and document independently, then compares vectors. Document embeddings are pre-computed, so searching a million chunks costs one query embedding plus an index lookup. Fast, scalable, approximate.
Cross-encoder — used for reranking
RecommendedFeeds query and document into the model together and outputs a single relevance score. The model sees the interaction between the two texts — which words in the document answer which part of the query. Accurate, and impossible to pre-compute (the score depends on the pairing), so it must run live, per candidate.
| Bi-encoder | Cross-encoder | |
|---|---|---|
| Sees query and document together | No | Yes |
| Pre-computable | Yes | No |
| Cost for 1M documents | One lookup | 1M model calls — impossible |
| Cost for 50 candidates | Trivial | ~50–200 ms — perfectly affordable |
| Relative accuracy | Good | Much better |
Why the two-stage design exists
Neither model can do the whole job. The bi-encoder can search millions but judges imprecisely; the cross-encoder judges precisely but cannot search millions. Chain them and you get the reach of the first with the judgement of the second. This recall-then-precision pattern recurs throughout information retrieval.
Reranker Options
| Type | How it works | Latency (50 docs) | Quality | Notes |
|---|---|---|---|---|
| Cross-encoder | Small transformer scores each pair | 50–200 ms | High | The standard choice |
| ColBERT / late interaction | Per-token vectors, matched at token level | 20–80 ms | High | Fast, but large storage cost |
| LLM reranker | Ask a language model to score or order candidates | 500–3,000 ms | Highest | Expensive; use for low-volume, high-value queries |
| Metadata reranking | Boost by recency, authority, source type | ~0 ms | Varies | Cheap; combine with the above |
A Note on LLM Rerankers
You can simply ask a capable model: "Rate how well this passage answers this question, 0–10." Quality is excellent — the model genuinely understands the relationship, including negation and subtle qualification.
The problem is cost and latency: 50 candidates means 50 model calls, or one very large call. Practical uses:
- Listwise reranking — one call ranking all candidates together, which also lets the model compare them against each other
- Only for high-value queries, routed there deliberately
- As a teacher — use LLM scores to generate training data for a cheap cross-encoder
Does Reranking Earn Its Place?
Usually yes, and by a wide margin
Reranking is one of the best value-per-effort upgrades in RAG. It typically adds 50–200 ms to a ~2,000 ms response — under 10% latency — while substantially improving which chunks reach the model. Compare that to chasing the same quality gain through better chunking or a better embedding model, both of which require re-indexing everything.
Reasons it might not pay off:
| Situation | Why |
|---|---|
| Retrieval is already near-perfect | Nothing left to reorder |
| Extreme latency constraints | Sub-500 ms total budgets leave no room |
| Very few candidates | Reranking 5 chunks has little to work with |
| Your real problem is upstream | Reranking cannot surface a chunk retrieval never returned |
A reranker can only reorder what retrieval gave it. If the correct chunk was not in the candidate set, reranking cannot help. When both recall and precision are poor, fix retrieval first — widen the candidate set, add hybrid search — and rerank after.
Context Assembly
You have the best 4 chunks. How you present them is not neutral.
The Lost-in-the-Middle Effect
Models attend unevenly across a long context. Information at the beginning and end is used reliably; information in the middle is measurably more likely to be overlooked — and the effect strengthens as the context grows.
Attention across a long context
Practical consequences:
| Rule | Reason |
|---|---|
| Put the highest-ranked chunk first | The most likely answer lands in a high-attention zone |
| Put the question last | Immediately before generation, in the other high-attention zone |
| Consider alternating placement (1st, 3rd, 4th, 2nd) | Pushes the two best chunks to both ends |
| Use fewer chunks | The shorter the context, the weaker the effect |
Deduplication
Overlapping chunks and duplicated source documents produce near-identical results. Two chunks conveying the same fact occupy two slots while adding one fact.
| Method | Catches |
|---|---|
| Exact hash | Identical text |
| Embedding similarity > ~0.95 | Near-duplicates and overlap regions |
| Substring containment | One chunk wholly inside another |
Context Compression
When retrieved chunks are long but only partly relevant, compress before generating.
| Technique | What it does | Trade-off |
|---|---|---|
| Extractive filtering | Keep only the sentences relevant to the query | Fast and safe; may drop useful surrounding context |
| Abstractive summarisation | Model rewrites chunks into a shorter form | Compresses hard; introduces a paraphrasing step that can itself distort facts |
| Sentence-window | Match one sentence, include N neighbours | Precise and cheap |
Abstractive compression puts a model between your source document and your answer. Any distortion it introduces is invisible downstream and breaks the guarantee that the answer traces to the source text. In regulated or high-stakes domains, prefer extractive methods.
Attribution Markers
Label each chunk so citation is mechanically possible:
[1] (anticoagulation-protocol.pdf, p.3)
Patients on warfarin should consult their prescriber before starting any NSAID.
[2] (drug-interactions.pdf, p.14)
NSAIDs including ibuprofen increase bleeding risk when taken with warfarin.Then instruct: "Cite the source number in square brackets after each claim." Without markers in the context, the model has nothing concrete to cite and will invent plausible-looking references.
A Complete Assembly Order
From candidates to prompt
Concept Checks
Check yourself
Why can't cross-encoder scores be pre-computed like embeddings?
Because the score is a property of the pair, not of the document. A cross-encoder reads query and document together and outputs how well that specific document answers that specific query — so you would need a score for every possible query, which is unbounded. Bi-encoder embeddings are pre-computable precisely because each document's vector is independent of any query.
Recall is poor AND ordering is poor. Do you add a reranker first?
No — fix retrieval first. A reranker only reorders the candidates it receives, so if the right chunk is not in the candidate set it cannot help at all. Widen the candidate set and add hybrid search to fix recall; then add reranking to fix ordering. Doing it the other way round produces a well-ordered list of wrong chunks.
Your answer misses a fact that was definitely in chunk 5 of 9. Likely cause?
Lost-in-the-middle. Chunk 5 of 9 sits in the weakest attention region. Two fixes, both cheap: retrieve fewer chunks so there is no deep middle, and order by rerank score so the strongest material occupies the start. If chunk 5 was genuinely the best chunk, that is also a signal your reranking is misordering.
Why is abstractive compression risky in a regulated domain?
Because it inserts a generative step between the source document and the answer. The compressed text is a model's paraphrase. If it distorts something, nothing downstream can tell the difference from real source content. Your citation now points at a document that does not actually say what was quoted. Extractive filtering preserves the original wording, so the audit trail holds.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Bi-encoder | Embeds separately; fast; used for search |
| Cross-encoder | Reads query and document together; accurate; used for reranking |
| Recall then precision | Retrieve wide with the fast model, narrow with the accurate one |
| Reranking cost | ~50–200 ms — under 10% of a typical response |
| Rerankers cannot recover | What retrieval never returned |
| Lost in the middle | Facts in the centre of a long context get overlooked |
| Ordering | Strongest chunk first, question last |
| Deduplication | Overlap steals slots without adding information |
| Extractive over abstractive | Preserves wording and the citation guarantee |
| Attribution markers | Without them, citations get invented |
Next
The context is assembled. Now make the model use it honestly: Generation & Grounding.
Retrieval Strategies
Dense, sparse, and hybrid search; BM25 and reciprocal rank fusion; query rewriting, expansion, HyDE, decomposition, and routing
Generation & Grounding
Prompting a model to answer only from retrieved context — citations, refusals, structured output, streaming, and hallucination control