What Is RAG?
The problem RAG solves, how it works, and how it compares to fine-tuning, long context, and web search — explained from zero
What Is RAG?
TL;DR
RAG stands for Retrieval-Augmented Generation. Instead of asking a Large Language Model (LLM) a question directly, you first search your own documents for relevant passages. Then you paste those passages into the prompt and ask the model to answer using only that text. The model supplies the language skill; your documents supply the facts.
| Property | Value |
|---|---|
| Level | Beginner — start here |
| Reading time | ~20 minutes |
| Prerequisites | None. If you know what a chatbot is, you can read this page. |
| You will understand | Why LLMs fail, what RAG fixes, and when not to use it |
First, Some Vocabulary
Every term in this section will be used constantly for the rest of the track. Nothing here assumes prior knowledge.
| Term | Expansion | Plain meaning |
|---|---|---|
| LLM | Large Language Model | A program trained on enormous amounts of text that predicts the next word. ChatGPT, Claude, and Gemini are interfaces to LLMs. |
| RAG | Retrieval-Augmented Generation | Search your documents first, then let the LLM answer using what was found. |
| Prompt | — | The complete block of text you send to the model. It includes your question and anything else you paste in. |
| Context window | — | The maximum amount of text a model can read at once. Measured in tokens. |
| Token | — | A chunk of text the model actually processes — roughly ¾ of an English word. "unbelievable" might be three tokens: un, believ, able. 100 tokens ≈ 75 words. |
| Corpus | — | Your whole collection of documents. Plural: corpora. |
| Chunk | — | One small piece of a document — usually a few hundred words — stored and retrieved as a unit. |
| Embedding | — | A list of numbers representing the meaning of a piece of text, so meaning can be compared with arithmetic. |
| Vector | — | A list of numbers. An embedding is a vector. The words are used interchangeably. |
| Grounding | — | Forcing the model to base its answer on supplied source text rather than its own memory. |
| Hallucination | — | When a model states something false with complete confidence. |
| Inference | — | One run of the model — one question in, one answer out. |
| Knowledge cutoff | — | The date after which the model's training data stops. It knows nothing later. |
Do not memorise this table. Every term is re-explained in context the first time it genuinely matters.
The Problem: Three Things an LLM Cannot Do
An LLM learned everything it knows during training, then that knowledge was frozen. This creates three hard limits.
Why a bare LLM is not enough
1. It has never seen your private data
The model was trained on public text. It has never read your company's support tickets, your patient records, your internal wiki, or the contract you signed last Tuesday. Ask about them and it must guess.
2. Its knowledge stops at a fixed date
Training data has a knowledge cutoff. Anything that happened afterwards — a policy change, a new drug warning, last quarter's numbers — simply does not exist to the model.
3. It cannot tell you when it does not know
This is the dangerous one. A model does not experience uncertainty the way you do. Asked something it has no information about, it will often produce a fluent, confident, completely fabricated answer — a hallucination. There is no warning label on the output.
What a Hallucination Actually Looks Like
Ask a model with no access to your data: "What is our refund window for enterprise customers?"
Enterprise customers receive a 45-day refund window, extendable to 60 days for annual contracts under section 7.2 of the standard agreement.
That answer is well-written, specific, cites a section number, and is entirely invented. The model was asked to produce plausible text, and it did exactly that. Fluency is not evidence of truth, and this is the single most important idea to internalise before building anything.
The Fix, In One Sentence
The core idea
Do not ask the model "what is our refund window?". Instead: find the paragraph in your documents that states the refund window, paste it into the prompt, and ask "using only the text below, what is our refund window?"
The model stops being a source of facts and becomes a reader of facts. That is the entire concept. Everything else in this track is engineering detail about how to find the right paragraph reliably, cheaply, and fast.
An Analogy
Imagine a brilliant new employee. Sharp, articulate, writes beautifully — but it is their first day and they have never seen your files.
- Without RAG: you ask them a question about company policy. Rather than admit ignorance, they improvise something that sounds right.
- With RAG: you hand them the three relevant pages from the handbook first, then ask. Now their intelligence works on your facts.
RAG is the act of handing over the right pages, automatically, for every question.
The Two Halves of Every RAG System
Every RAG system has exactly two pipelines. Confusing them is the most common beginner mistake, so hold on to this distinction.
The two pipelines
Indexing pipeline (offline)
Runs ahead of time, once per document. Load → clean → chunk → embed → store. Slow is fine; nobody is waiting.
Query pipeline (online)
Runs every time someone asks a question. Embed question → search → rerank → build prompt → generate. Must be fast; a person is watching a spinner.
| Indexing pipeline | Query pipeline | |
|---|---|---|
| When it runs | Ahead of time, in batches | On every single question |
| Triggered by | A new or changed document | A user |
| Speed requirement | Minutes or hours are acceptable | Milliseconds to a few seconds |
| Cost profile | Paid once per document | Paid per question, forever |
| If it breaks | Search quality quietly degrades | The user sees an error |
Roughly 80% of RAG quality problems originate in the indexing pipeline — bad parsing, bad chunking, missing metadata — but they only appear as bad answers in the query pipeline. When retrieval seems broken, look at what you stored before you touch the search code.
Watch It Happen
Before the detail, see the whole query pipeline run once. Press play.
The query pipeline, step by step
Question
+0 msThe user asks something in plain language.
Data leaving this stage
"Can I take ibuprofen with warfarin?"
Nothing has happened yet. This raw string is all the system has to work with.
Look at the timing bar: generation dominates everything else. Retrieval and reranking together cost less than a fifth of the total, which is why adding a reranker is usually a cheap upgrade in latency terms.
Every stage in that animation gets a dedicated page later in this track. If a stage does not make sense yet, that is expected — you have now seen the shape of the whole thing, which makes each part easier to place.
RAG vs The Alternatives
RAG is not the only way to give a model knowledge. Knowing when not to use it matters as much as knowing how.
RAG vs Fine-Tuning
Fine-tuning means continuing to train the model on your own examples, permanently changing its internal weights.
RAG or fine-tuning?
Use RAG for knowledge — facts the model must be correct about
RecommendedFacts change; weights are expensive to change. Adding a document to a RAG system takes seconds. Updating a fine-tuned model means retraining. RAG also gives you citations, which fine-tuning cannot: a fine-tuned model absorbs facts into its weights, so there is no source to point at.
Use fine-tuning for behaviour — format, tone, and style
If the model keeps answering in the wrong tone, ignoring your output format, or failing at a specialised task pattern, no amount of retrieved context fixes that reliably. Teach behaviour with examples in the weights.
Using both together is normal
Fine-tune for the house style and output structure; retrieve for the facts. They solve different problems and do not compete.
| Question | RAG | Fine-tuning |
|---|---|---|
| Time to add new knowledge | Seconds | Hours to days (retrain) |
| Upfront cost | Low | High (compute + labelled data) |
| Cost per query | Higher (extra tokens for context) | Lower (no context to send) |
| Can cite sources | Yes | No |
| Can revoke a document | Yes — delete it from the index | No — it is baked into the weights |
| Handles frequently changing data | Excellent | Poor |
| Changes the model's writing style | Weakly | Strongly |
| Needs labelled training examples | No | Yes |
RAG vs Just Using a Long Context Window
Modern models accept very large context windows — hundreds of thousands of tokens. A fair question: why not skip retrieval and paste in everything?
| Consideration | Reality |
|---|---|
| Cost | You pay per token on every question. Sending 200,000 tokens to answer "what are your opening hours?" is enormously wasteful — and you pay it again on the next question. |
| Latency | More input tokens means a slower response. Long contexts add seconds. |
| Scale | Corpora are usually far bigger than any context window. 10,000 documents will never fit, no matter how large the window grows. |
| Accuracy | Models attend unevenly across a long context. Facts buried in the middle of a huge document are measurably more likely to be missed — the "lost in the middle" effect. Less, better-targeted context often beats more context. |
Long context and RAG are complements, not rivals. A common modern design retrieves fewer, larger chunks — whole sections rather than paragraphs — because the window can afford them. The Long Context RAG project builds exactly this.
The Full Decision Table
| Your situation | Best tool |
|---|---|
| Facts live in your private documents | RAG |
| Facts change weekly or daily | RAG |
| You must show where an answer came from | RAG |
| You need a specific output format or tone | Fine-tuning |
| The task is narrow, repetitive, and high-volume | Fine-tuning (cheaper per call) |
| The whole corpus is small and static (a few pages) | Just paste it into the prompt — no RAG needed |
| The answer requires live data (stock price, order status) | Tool calling / an API, not RAG |
| The answer needs public web information | Web search as a tool |
Do not build RAG when a simpler design works. If your entire knowledge base is a 5-page FAQ, put it in the system prompt and stop. RAG adds an index, an embedding model, a database, and several new failure modes. Earn that complexity before you pay for it.
A Concrete Walkthrough
Follow one real question through a clinic assistant, end to end.
Setup (indexing, ran last night): 400 clinic documents were loaded, split into 6,200 chunks, embedded, and stored.
The question arrives: "Can I take ibuprofen with my warfarin?"
One question, end to end
What the model actually receives — this is the part beginners rarely see, so read it closely:
You are a clinical information assistant. Answer the question using ONLY the
context below. If the context does not contain the answer, say
"I don't have that information" — do not use outside knowledge.
CONTEXT:
[1] (drug-interactions.pdf, p.14) Non-steroidal anti-inflammatory drugs
including ibuprofen increase bleeding risk when taken with warfarin,
both by inhibiting platelet function and by irritating the gastric lining.
[2] (anticoagulation-protocol.pdf, p.3) Patients on warfarin should consult
their prescriber before starting any NSAID. Paracetamol is generally
preferred for mild pain in anticoagulated patients.
QUESTION: Can I take ibuprofen with my warfarin?The model never sees the other 6,196 chunks. It does not need to. Retrieval reduced 400 documents to two paragraphs, and everything the answer needs is in those two paragraphs.
What RAG Does Not Fix
Being clear-eyed about the limits prevents a lot of wasted effort.
| Misconception | Reality |
|---|---|
| "RAG eliminates hallucination" | It greatly reduces it. A model can still misread supplied context, blend it with training knowledge, or over-generalise. Grounding is a strong defence, not a guarantee. |
| "RAG means the answer is correct" | RAG guarantees the answer is based on your documents. If your documents are wrong or outdated, RAG faithfully reproduces the error. |
| "Better prompts will fix bad retrieval" | If the relevant chunk was never retrieved, no prompt can recover it. The model cannot read what it was not given. |
| "RAG works out of the box" | A naive implementation typically retrieves the right chunk 60–75% of the time. Production systems reach 90%+ only through chunking, hybrid search, reranking, and evaluation — the rest of this track. |
| "More retrieved chunks means better answers" | Past a point, extra chunks add noise, cost tokens, and reduce accuracy by diluting the signal. |
Where RAG Is Used
| Domain | Application | Why RAG fits |
|---|---|---|
| Customer support | Answer from help-centre articles and past tickets | Content changes constantly; agents need citations |
| Healthcare | Search clinical guidelines and literature | Answers must be traceable to a source |
| Legal | Contract review, clause lookup | Documents are private, long, and precise wording matters |
| Finance | Filings, earnings calls, research notes | Recency is essential; auditability is mandatory |
| Engineering | Codebase and internal documentation search | Private, huge, and updated hourly |
Every one of these has a worked case study on this site — see the RAG case studies.
Concept Checks
Answer these out loud before moving on. If any answer is fuzzy, re-read the linked section.
Check yourself
Why can't a better prompt fix a retrieval failure?
Because the prompt only shapes how the model uses what it was given. If the chunk containing the answer never made it into the context, the information is simply absent. Prompt engineering operates downstream of retrieval, and no downstream stage can recover data lost upstream.
You need the model to always reply in formal British English AND to know today's product prices. What do you use?
Both tools. Fine-tuning (or at minimum a strong system prompt) for the language style, because that is behaviour. RAG for the prices, because those are facts that change and must be current and citable.
Your corpus is 12,000 tokens total and never changes. Should you build RAG?
No. It fits comfortably in a modern context window. Paste it into the system prompt. Building an index, an embedding pipeline, and a vector database for 12,000 static tokens adds cost and failure modes with no benefit.
Why does RAG support citations when fine-tuning cannot?
RAG retrieves discrete passages that still exist as identifiable objects at answer time, so each claim can point back at a document and page. Fine-tuning dissolves information into billions of weight values — there is no longer any retrievable "source" to point at.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| RAG | Search your documents, then let the model answer using only what was found |
| Hallucination | Confident, fluent, false output — the failure RAG primarily targets |
| Grounding | Constraining the answer to supplied source text |
| Token | The unit models read and you are billed for; ~¾ of a word |
| Context window | How much text a model can read in one call |
| Indexing pipeline | Offline: load → chunk → embed → store |
| Query pipeline | Online: embed → search → rerank → prompt → generate |
| RAG vs fine-tuning | RAG for knowledge and citations; fine-tuning for behaviour and format |
| RAG vs long context | Complementary — retrieval controls cost, latency, and scale |
| The hard limit | The model cannot read what it was not given |
Next
Now that you know what RAG is, take apart the machine: The RAG Pipeline, Stage by Stage.
Prefer to build first and read after? Go straight to Intelligent Document Q&A and return here when a term puzzles you.