Document Loading & Parsing
Turning PDFs, HTML, tables, and scans into clean text — the stage that silently caps the quality of everything downstream
Document Loading & Parsing
TL;DR
Before anything can be chunked, embedded, or searched, it must become text. This stage is boring, thankless, and the most common hidden cause of bad RAG. A PDF is not a text file — it is a page-drawing program — and extracting correct reading order, tables, and headings from one is real engineering. What you fail to extract here is permanently invisible to your system.
| Property | Value |
|---|---|
| Level | Beginner → Intermediate |
| Reading time | ~20 minutes |
| Prerequisites | The RAG Pipeline |
| You will understand | Why PDFs are hard, how to handle tables and scans, and what metadata to capture |
Why This Stage Decides Your Ceiling
Your system can never answer a question about text it failed to extract. Maybe a table was flattened into a row of meaningless numbers. Maybe a two-column page came out as scrambled sentences. Maybe a scanned appendix produced nothing at all. As far as retrieval is concerned, those facts do not exist. No embedding model, reranker, or prompt recovers them.
This makes loading a hard ceiling on quality. Everything downstream can only lose information, never add it.
Information can only be lost, never regained
Vocabulary
| Term | Expansion | Meaning |
|---|---|---|
| Parsing | — | Converting a file format into structured text |
| OCR | Optical Character Recognition | Reading text out of an image, pixel by pixel |
| Reading order | — | The sequence a human would read blocks of a page in |
| Layout analysis | — | Detecting which regions are headings, body, tables, captions |
| Boilerplate | — | Repeated non-content: headers, footers, nav bars, cookie banners |
| Metadata | — | Structured facts about the document: source, page, date, author |
| Provenance | — | The trail showing exactly where a piece of text came from |
Format by Format
How hard is each format, really?
Plain text and Markdown — trivial
Read the file. Markdown is a gift: # headings give you document structure for free, which you should preserve for chunking and for context prefixes.
HTML — easy, but noisy
Extraction is simple; the problem is that ~60% of a typical web page is navigation, ads, cookie banners, and footers. Strip to the main content region. Preserve heading tags and table structure — that structure is expensive to reconstruct later.
Word and Office documents — medium
Structure is available in the file format, which helps. Watch for tracked changes and comments leaking into the body text, and for content buried in text boxes that many parsers skip entirely.
Digital PDF — genuinely hard
A PDF stores instructions like "draw glyph 'A' at coordinate (72, 340)". There is no paragraph, no reading order, often not even a word — those must be inferred from position. Multi-column layouts, footnotes, and headers routinely come out interleaved.
Scanned PDF and images — very hard
There is no text at all, only pixels. You need OCR, and OCR makes mistakes: 1 becomes l, 0 becomes O, rn becomes m. Quality depends on scan resolution, skew, and contrast. Always sample the output before trusting it.
Slides and spreadsheets — awkward
In slides, meaning lives in layout, images, and speaker notes rather than in sentences. In spreadsheets, meaning lives in the relationship between a cell and its row and column headers — extract naively and you get a wall of context-free numbers.
The Two-Column PDF Problem
This is the classic failure and worth seeing concretely. A page laid out in two columns:
┌─────────────────┬─────────────────┐
│ Warfarin dosing │ Contraindicat- │
│ begins at 5 mg │ ions include │
│ daily, adjusted │ active bleeding │
│ to INR. │ and pregnancy. │
└─────────────────┴─────────────────┘A naive parser reads left to right across the whole page, producing:
"Warfarin dosing Contraindicat- begins at 5 mg ions include daily, adjusted active bleeding to INR. and pregnancy."
Now embed that. It means nothing, it matches nothing, and the two real facts it contained are gone. A layout-aware parser detects the column boundary and reads each column fully before moving on.
Test for this in thirty seconds: extract one multi-column document and read the output. If sentences interleave, your parser is not layout-aware and you need a better one. The Document RAG with Docling project builds a full layout-aware pipeline.
Tables Deserve Their Own Section
Tables carry a disproportionate share of the facts people actually ask about — prices, doses, limits, dates, comparisons — and they are the most commonly destroyed structure in RAG.
What Goes Wrong
Flattening this table:
| Drug | Starting dose | Max dose |
|---|---|---|
| Warfarin | 5 mg | 10 mg |
| Apixaban | 5 mg | 10 mg |
...into naive text produces Drug Starting dose Max dose Warfarin 5 mg 10 mg Apixaban 5 mg 10 mg. Ask "what is the maximum dose of apixaban?" and the retrieved text contains four dose numbers with no reliable way to bind one to a drug. The model must guess, and it may guess wrong — confidently.
How to Handle Tables Properly
Table strategies, worst to best
Flatten into a text blob
Loses every row-column relationship. Avoid.
Preserve as Markdown or HTML
Keeps the grid visible, and modern models read Markdown tables well. Good baseline. Weakness: a large table split by chunking loses its header row, orphaning the remaining rows.
Repeat headers into every row
Convert each row into a self-contained sentence: "Warfarin: starting dose 5 mg, maximum dose 10 mg." Now every row survives chunking independently and retrieves precisely.
Table summary plus structured original
RecommendedStore a natural-language summary of the table for retrieval (embeddings match prose better than grids), and keep the structured original for the model to read when that summary is retrieved. Best accuracy; slightly more storage.
If a table is split by chunking, every chunk after the first loses the header row. Rows become Apixaban | 5 mg | 10 mg with no indication of what those columns mean. Either keep tables whole, or repeat headers per row.
Metadata: Capture It Now or Lose It Forever
Metadata is cheap to capture during loading and expensive-to-impossible to reconstruct later. Capture generously.
| Field | Why it matters | Cost of missing it |
|---|---|---|
source_file | Citation, deduplication | Cannot tell users where an answer came from |
page_number | Citation precision | "It's somewhere in this 200-page PDF" |
section_heading | Context prefixes, filtering | Chunks lose their topic |
created_at / updated_at | Freshness filtering, recency ranking | Cannot prefer current policy over the 2019 version |
document_type | Routing and filtering | Cannot search "only contracts" |
author / department | Filtering, authority weighting | No way to prefer official sources |
access_level | Security | Users see documents they must not see |
language | Model routing | Wrong-language results pollute retrieval |
version | Supersession | Old and new policy both retrieved; model averages them |
Access control is a loading-stage concern. Attach permission metadata to every chunk at index time. Without it you cannot filter results per user at query time, and your RAG system turns into a very efficient tool for leaking confidential documents. Adding this later means re-indexing everything.
Cleaning
Once you have text, remove what is not content.
| Operation | Removes | Why |
|---|---|---|
| Boilerplate stripping | Repeating headers/footers, nav, cookie banners | Otherwise every chunk contains the same noise and looks similar to every query |
| Whitespace normalisation | Runs of spaces, newlines, tabs | Wastes tokens; disrupts embeddings |
| Encoding repair | ’, é, mojibake | Corrupted characters degrade embedding quality |
| Unicode normalisation | Smart quotes, ligatures, non-breaking spaces | Prevents exact-match keyword search failures |
| De-hyphenation | anti-\ncoagulant → anticoagulant | Line-break hyphens split words that should match |
| Page-number removal | Bare - 14 - lines | Pure noise |
Do not over-clean
Resist the urge to strip punctuation, lowercase everything, or remove stop words. Those are habits from older keyword-search systems. Modern embedding models are trained on natural text and use case, punctuation, and function words as signal. Aggressive normalisation actively hurts embedding quality.
Deduplication
Real corpora are full of near-duplicates: the same policy in three folders, an email quoted in twelve replies, boilerplate legal text in every contract.
Duplicates hurt in a specific way: they occupy multiple slots in your top-k results. Retrieve 5 chunks, and if 4 are copies of the same paragraph, you effectively retrieved 2 distinct facts instead of 5. Deduplicate at index time by content hash for exact copies, and by embedding similarity above a threshold (~0.97) for near-copies.
A Practical Loading Recipe
# Sketch of a loading stage. The specific libraries matter less than
# the shape: extract with layout awareness, capture metadata at the
# same moment, clean conservatively, and always inspect the output.
def load_document(path):
# 1. Extract with a layout-aware parser, not a naive text dump.
parsed = layout_aware_parse(path) # detects columns, tables, headings
# 2. Capture metadata NOW — page numbers are unrecoverable later.
base_meta = {
"source_file": path.name,
"document_type": detect_type(path),
"updated_at": path.stat().st_mtime,
"access_level": lookup_permissions(path), # never skip this
}
sections = []
for block in parsed.blocks:
if block.is_table:
# Rows as self-contained sentences survive chunking intact.
text = table_to_sentences(block)
else:
text = clean(block.text) # whitespace, encoding, de-hyphenation
if not text.strip():
continue
sections.append({
"text": text,
"metadata": {
**base_meta,
"page_number": block.page,
"section_heading": block.nearest_heading,
"is_table": block.is_table,
},
})
return sectionsBefore you write another line of pipeline code, print the output of this stage and read it. Not the length, not the chunk count — the actual text. Ten minutes of reading here routinely saves days of debugging retrieval that was never the problem.
Concept Checks
Check yourself
Your system answers well from Word documents but poorly from PDFs of the same content. Where do you look?
The PDF parser, first and only. Same content, different format, different results — the variable is extraction. Print the extracted PDF text and look for interleaved columns, missing tables, or dropped scanned pages. Do not touch chunk size or the embedding model.
Why is 'maximum dose of apixaban' a hard question for a flattened table?
Because flattening destroys the row-column binding. The retrieved text contains several drugs and several dose figures with no structural link between them. The model has to infer which number belongs to which drug from word order alone, and it can get that wrong while sounding certain.
Why is skipping access_level metadata a security bug rather than a feature gap?
Because retrieval has no other way to know a user may not see a document. Without that field, filtering is impossible and the system will happily retrieve and quote confidential content to anyone who asks a matching question. It is also expensive to fix — adding the field later requires re-indexing the whole corpus.
Why not lowercase and strip punctuation like a classic search engine would?
Because embedding models are trained on natural text and use case and punctuation as meaning signals. Those normalisation steps were designed for exact-token matching systems. Applied to embeddings, they discard information the model was trained to use, and measurably reduce retrieval quality.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Loading sets the ceiling | Downstream stages can only lose information, never recover it |
| PDFs are drawing instructions | Reading order and structure must be inferred, not read |
| Layout awareness | Required for multi-column documents; test it explicitly |
| OCR | Needed for scans; introduces character-level errors |
| Tables | Repeat headers per row, or store a summary plus the structured original |
| Metadata at load time | Page, section, date, version, and access level — unrecoverable later |
| Access control metadata | A security requirement, not a nice-to-have |
| Conservative cleaning | Strip boilerplate and fix encoding; do not lowercase or de-punctuate |
| Deduplication | Duplicates steal slots in your top-k results |
| Read your output | The cheapest debugging step in the entire pipeline |
Next
You have clean text. Now decide how to cut it up — the highest-leverage choice in RAG: Chunking Strategies.
The RAG Pipeline, Stage by Stage
Every stage of the indexing and query pipelines — what it does, what it outputs, what it costs, and how it fails
Chunking Strategies
How to split documents — fixed, recursive, semantic, parent-document, and late chunking — and why boundaries decide what your system can ever answer