Multimodal RAG
Retrieving from images, tables, charts, and scanned pages — captioning, multimodal embeddings, vision models, and picking between them
Multimodal RAG
TL;DR
Real documents are not walls of text. They hold tables, charts, diagrams, screenshots, and photographs. A text-only pipeline throws all of that away silently — the page still processes, it just answers as if the picture was never there. Multimodal RAG means making non-text content searchable too. There are three ways to do it, and they suit different problems.
| Property | Value |
|---|---|
| Level | Intermediate → Advanced |
| Reading time | ~22 minutes |
| Prerequisites | Conversational RAG |
| You will understand | Captioning, multimodal embeddings, vision models, and how to handle tables |
What Gets Lost
Take a normal company report. A text-only pipeline extracts the paragraphs and drops everything else:
| Content | What a text-only pipeline sees |
|---|---|
| A revenue chart | Nothing. Maybe the caption. |
| A comparison table | A jumble of numbers with no rows or columns |
| An architecture diagram | Nothing |
| A scanned contract page | Nothing — a scan is a picture, not text |
| A product photo | Nothing |
| A screenshot with an error message | Nothing |
The dangerous part is that nothing appears to break. No error is raised. Documents process, answers come back, and everything looks fine — while the numbers your finance team actually cares about, which only ever existed inside a chart, are invisible to the system. Nobody finds out until someone asks a question about that chart.
The Three Approaches
Three ways to make images searchable
1. Describe it in text
A vision model writes a description of each image. You store and search that text.
2. Embed images directly
A model that understands both text and images puts them in one shared vector space.
3. Feed pages to a vision model
Skip extraction. Send the page image itself to a model that can read it.
Approach 1 — Describe images as text
The simplest and most common method. A vision model looks at each image and writes a description. That description is embedded and stored like any other chunk.
[Image: quarterly-revenue.png]
↓ vision model
"A bar chart showing quarterly revenue for 2024. Q1 is 4.2 million,
Q2 is 5.1 million, Q3 is 4.8 million, and Q4 is 6.3 million.
The trend rises overall, with a dip in Q3."
↓ embed and store as an ordinary chunk| Aspect | Detail |
|---|---|
| Strength | Works with your existing text pipeline. Nothing else changes. |
| Strength | The description is readable, so you can check it and debug it |
| Weakness | You only keep what the description mentions. Anything left out is gone forever. |
| Cost | One vision model call per image, paid once at indexing time |
Give the vision model instructions about your domain. "Describe this image" produces vague output. "Describe this chart. State every axis label and every data value you can read." produces something actually searchable. This one change makes the biggest difference in the whole approach.
Approach 2 — Embed images directly
Some models embed text and images into the same vector space. A photograph of a dog and the words "a dog" land near each other. This means you can search images with a text query, with no description step in between.
| Aspect | Detail |
|---|---|
| Strength | No information lost to a written description |
| Strength | Genuinely good at "find the picture that looks like this" |
| Weakness | Weaker on fine detail — it will not read the small numbers on a chart |
| Weakness | Harder to debug. You cannot read a vector to see what it captured. |
| Best for | Photo libraries, product catalogues, visual similarity search |
Approach 3 — Send the page to a vision model
Skip text extraction entirely. Store page images, retrieve whole pages, and let a model that can see read them directly.
| Aspect | Detail |
|---|---|
| Strength | Nothing is lost, because nothing was extracted. Layout, tables, and charts stay intact. |
| Strength | Excellent for complex layouts that parsers mangle |
| Weakness | Expensive — page images cost far more tokens than text |
| Weakness | You still need a way to find the right page in the first place |
| Best for | Scanned documents, forms, dense financial reports |
Choosing Between Them
Which approach fits your documents?
Reports, papers, and manuals with some charts
RecommendedDescribe images as text. It plugs into what you already have, it is cheap after indexing, and you can read the descriptions to check they are right.
A photo or product catalogue
Embed images directly. Visual similarity is the actual task here, and descriptions would throw away exactly the detail you need.
Scanned forms and contracts
Send pages to a vision model, or use a strong parser with OCR. Text extraction on scans is where most pipelines fail hardest.
Mostly text, a few decorative images
Do nothing. If the pictures carry no information, processing them costs money and adds noise. Check first whether anyone actually asks about them.
Many real systems combine approaches: descriptions for charts and diagrams, direct page images for scanned pages, and plain text for everything else. Route by content type at indexing time.
Tables Deserve Their Own Section
Tables are the most common non-text content, the most valuable, and the most often broken. They are worth handling separately.
The problem: a table flattened into a line of text loses the link between a number and its row and column.
BAD — flattened, meaning destroyed:
Region Q1 Q2 Q3 North 120 145 138 South 98 102 115
Which number belongs to which region? Unrecoverable.GOOD — one row per chunk, each standing alone:
"North region: Q1 revenue 120, Q2 revenue 145, Q3 revenue 138."
"South region: Q1 revenue 98, Q2 revenue 102, Q3 revenue 115."| Technique | When to use |
|---|---|
| One row per chunk, written as a sentence | Small and medium tables. Each row becomes retrievable on its own. |
| Keep the table whole with a summary above it | Small tables where rows must be compared |
| Store the table separately and query it as data | Large tables. Retrieval finds the table; a real query engine answers the question. |
| Keep the header with every chunk | Always. Without column names the numbers mean nothing. |
Never let a table split across a chunk boundary without repeating the header. Half a table with no column names is worse than no table at all, because it produces confident answers built on numbers the model has misread.
What Changes in the Pipeline
Multimodal indexing
Input
Sort by type
Handle each type
Store together
At answer time
Two things matter here:
| Rule | Why |
|---|---|
| Record the content type in metadata | Lets you filter to tables only, or debug one type at a time |
| Always keep a link back to the original image | So you can show the user the real chart next to the answer. A description is for searching, not for proof. |
Common Mistakes
What goes wrong
Assuming a scanned PDF contains text
A scan is a picture of a page. Text extraction returns nothing, silently. Always check whether your PDFs have a text layer before trusting the pipeline.
Flattening tables into plain text
Numbers lose their rows and columns and become meaningless. The model then reads them confidently and wrongly.
Vague image descriptions
"A chart showing data" is unsearchable. Ask for axis labels and actual values.
Showing the description instead of the image
Users need to see the real chart to trust the answer. Keep the link to the original.
Processing every image regardless of value
Logos, borders, and decorative photos cost money to describe and add noise to retrieval. Filter by size and position first.
Not recording the content type
You lose the ability to filter or to debug one modality at a time.
Concept Checks
Check yourself
Why is a text-only pipeline on image-heavy documents so dangerous?
Because it fails silently. No error appears, documents index normally, and answers keep coming back. The system simply behaves as though the charts and diagrams do not exist. If the important numbers live only inside a chart, the pipeline will confidently answer from surrounding prose instead, and nobody discovers the gap until a user asks about the chart directly.
When would you describe images as text rather than embed them directly?
When the information is in the detail — numbers, labels, and axis values — and when you want to check what was captured. A description records "Q4 is 6.3 million" as searchable text you can read and verify. Direct image embeddings capture overall visual character well but will not reliably preserve small printed values, and you cannot inspect a vector to find out what it missed.
Why must the header row be repeated on every table chunk?
Because a number without its column name carries no meaning. A chunk reading "North 120 145 138" gives the model no way to know these are quarterly revenue figures, or which quarter is which. It will guess, and it will sound certain. Repeating the header costs a few tokens per chunk and is the difference between usable data and confident errors.
You describe images for search. Why still keep the original?
Because the description is a search tool, not evidence. A user asking about revenue trends should see the actual chart next to the answer so they can check it themselves. The description is also lossy by nature — showing the original lets the user recover anything the description left out, which is exactly the failure mode this approach has.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| The silent failure | Text-only pipelines drop images without any error |
| Describe as text | Vision model writes a description; you search that. The usual default. |
| Direct image embedding | Text and images in one vector space. Best for visual similarity. |
| Page as image | Send the page picture to a vision model. Best for scans and complex layouts. |
| Detailed prompts | "State every axis label and value" beats "describe this image" |
| Tables need row-level chunks | One row per chunk, written as a sentence |
| Repeat the header | Always. Numbers without column names are meaningless. |
| Record content type | Enables filtering and per-type debugging |
| Keep the original | Descriptions are for finding; images are for proving |
Next
Back to the bigger picture — the architectures built on top of all this: RAG Architectures.
Conversational RAG
Making RAG work across many turns — rewriting follow-up questions, managing chat history, resolving references, and handling topic changes
RAG Architectures
Naive, Advanced, and Modular RAG — plus HyDE, RAG-Fusion, Self-RAG, CRAG, Adaptive, Agentic, Graph, and Speculative RAG, and how to choose between them