Conversational RAG
Making RAG work across many turns — rewriting follow-up questions, managing chat history, resolving references, and handling topic changes
Conversational RAG
TL;DR
Everything so far assumed one question, asked once. Real users chat. They ask "what about the second one?" and "is that still true for Europe?" — questions that mean nothing on their own. Sending them straight to a search engine finds nothing useful. The fix is to rewrite each follow-up into a full, standalone question before you search.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~20 minutes |
| Prerequisites | Generation & Grounding |
| You will understand | Query rewriting, history management, and why chat breaks normal RAG |
Why One-Shot RAG Breaks in a Chat
Look at a short conversation:
User: What is our refund window for hardware?
Bot: 30 days from delivery. [warranty-policy.pdf]
User: What about software?Now search for "What about software?". Think about what those four words contain. No mention of refunds. No mention of a time window. The search returns whatever documents happen to discuss software — release notes, licence terms, install guides. None of them answer the question.
The user's question was perfectly clear to a human, because a human carries the previous turn in their head. Your search engine does not. It sees four words with no useful content. This is the single biggest cause of bad answers in chat interfaces.
The Fix: Rewrite Before You Search
Add one step in front of retrieval. Take the chat history plus the new message, and produce a question that stands on its own.
Conversational RAG
Note that steps 3 and 4 are unchanged from ordinary RAG. Conversational RAG is not a new architecture. It is one extra step at the front.
What the rewriter is doing
| Job | Before | After |
|---|---|---|
| Fill in the missing topic | "What about software?" | "What is the refund window for software?" |
| Resolve pronouns | "Does it apply in Europe?" | "Does the 30-day refund window apply in Europe?" |
| Resolve references | "Tell me about the second one" | "Tell me about the extended warranty plan" |
| Keep the question whole | "And the cost?" | "What is the cost of the extended warranty plan?" |
Give the rewriter the last 3–5 turns, not the whole conversation. More history makes it slower, more expensive, and worse — it starts dragging in old topics that the user has moved on from.
Knowing When Not to Rewrite
This is the part people miss. Some messages are already complete, and rewriting them causes harm.
Rewrite or leave alone?
"What about software?"
Rewrite. Meaningless on its own. This is exactly the case rewriting exists for.
"How do I reset my password?"
Leave it alone. It is already complete. A rewriter given earlier turns about refunds may glue them on and produce "How do I reset my password for a refund?" — which finds nothing.
"Actually, forget that. What are your office hours?"
Leave it alone, and clear the history. The user has changed subject deliberately. Carrying the old topic forward actively fights them.
"Thanks!"
Do not search at all. There is no question here. Retrieving for social messages wastes money and pollutes the context with random chunks.
The practical answer is to let the rewriter decide. Instruct it: "If the message already stands on its own, return it unchanged. If it is not a question, say NO_SEARCH."
Managing the History
Conversations grow. Context windows do not grow with them. You need a policy.
| Strategy | How it works | Trade-off |
|---|---|---|
| Sliding window | Keep the last N turns, drop the rest | Simple and fast. Forgets things said early on. |
| Summary | Keep a running summary of the whole chat | Remembers old facts. Costs a model call per turn, and details get lost. |
| Summary + window | A summary of old turns, plus the last few turns in full | The usual choice. Recent detail stays exact, older context survives in outline. |
| Full history | Send everything | Only works for short chats. Cost and latency climb every turn. |
Watch your token count per turn, not just per query. A chat that works fine at turn 3 can quietly become slow and expensive by turn 20, because the history keeps growing while nobody is looking at it.
Two separate uses of history
Keep these apart in your head — they need different amounts of history:
| Used for | How much history | Why |
|---|---|---|
| Rewriting the question | Last 3–5 turns | Only recent context is needed to resolve "it" or "that one" |
| Generating the answer | More, plus the summary | The model needs the conversation's flow to answer in a natural way |
Retrieval Across Turns
Should you search again on every turn?
| Situation | Do this |
|---|---|
| New topic | Search fresh |
| Follow-up on the same documents | Reuse the chunks you already retrieved — faster and cheaper |
| Follow-up needing more detail | Search again with the rewritten question |
| Pure social message | Do not search |
A simple, effective rule: search on every turn where the rewriter changed the question or produced a new topic, and reuse otherwise.
Citations in a Conversation
Citations get harder across turns. A claim made at turn 2 may be referred to at turn 8, long after those chunks left the context.
| Problem | Fix |
|---|---|
| Model cites a source from an earlier turn that is no longer in context | Keep a running list of sources used in the chat, with stable IDs |
| Same document cited under different numbers on different turns | Use stable document IDs, not per-turn numbering |
| User asks "where did you get that?" about an old answer | Store the sources with each answer in the chat log |
Common Mistakes
What goes wrong in chat RAG
Searching with the raw message
The default failure. "What about software?" retrieves nothing useful, and the model then answers from training data or refuses. Rewriting is not optional in a chat interface.
Feeding the whole history to the rewriter
It starts merging old topics into new questions. Three to five turns is enough.
Rewriting messages that are already complete
Turns a good question into a worse one by attaching stale context. Let the rewriter return the message unchanged.
Never clearing history on a topic change
The user moved on; your pipeline did not. Detect subject changes and reset.
Ignoring token growth
Turn 20 costs several times what turn 2 cost, and nobody notices until the bill arrives.
Retrieving on greetings and thanks
Wastes a search and a generation, and drops unrelated chunks into the context.
Concept Checks
Check yourself
Why does searching with the raw follow-up message fail so badly?
Because the message often has almost no content of its own. "What about software?" contains no mention of refunds or time windows, so a search engine matching on meaning or on words has nothing useful to match. The meaning lived in the previous turn, which the search engine never saw. A human fills that gap automatically, which is why the bug is easy to miss when you test by hand.
Why limit the rewriter to the last few turns instead of the whole chat?
Because older turns are usually about different topics, and the rewriter cannot always tell which parts still matter. Given a long history it tends to blend old subjects into the new question, producing something that matches nothing. Recent turns hold everything needed to resolve "it" or "that one", so more history adds cost and noise without adding accuracy.
A user says 'Actually, forget that — what are your office hours?' What should happen?
The message is already complete, so it should go to search unchanged, and the earlier history should be dropped. This is a deliberate topic change. Rewriting it against the previous subject would attach context the user has explicitly abandoned, and the search would then look for office hours mixed with refunds. Detecting these resets is as important as doing the rewriting.
Why keep a running list of sources for the whole conversation?
Because users refer back to earlier answers, and by then those chunks have left the context window. If the model is asked "where did that come from?" at turn 8 about something said at turn 2, it has nothing to work from and will invent a source. Storing the sources used with each answer means the reference can be looked up rather than guessed.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| The core problem | Follow-up questions mean nothing on their own |
| Query rewriting | Turn the follow-up into a standalone question before searching |
| One extra step | Retrieval and generation are unchanged; only the front changes |
| Do not always rewrite | Complete questions and topic changes should pass through untouched |
| NO_SEARCH | Greetings and thanks need no retrieval at all |
| Summary plus window | The usual history strategy — recent turns exact, old turns summarised |
| Different history sizes | 3–5 turns for rewriting; more for generating |
| Reuse chunks | Follow-ups on the same documents need no fresh search |
| Stable source IDs | So citations survive across turns |
| Watch token growth | Turn 20 costs far more than turn 2 |
Next
Chat handled. Now the other thing real documents contain — pictures, tables, and charts: Multimodal RAG.
Generation & Grounding
Prompting a model to answer only from retrieved context — citations, refusals, structured output, streaming, and hallucination control
Multimodal RAG
Retrieving from images, tables, charts, and scanned pages — captioning, multimodal embeddings, vision models, and picking between them