Chunking Strategies
How to split documents — fixed, recursive, semantic, parent-document, and late chunking — and why boundaries decide what your system can ever answer
Chunking Strategies
TL;DR
Chunking is splitting documents into the small pieces you store and retrieve. It is the highest-leverage decision in RAG: a boundary that lands in the middle of a fact destroys that fact permanently. This page covers every common strategy, how to pick chunk size and overlap, and the advanced patterns that escape the size trade-off entirely.
| Property | Value |
|---|---|
| Level | Beginner → Advanced |
| Reading time | ~25 minutes |
| Prerequisites | Document Loading |
| You will understand | Every chunking strategy, when to use each, and how to tune size and overlap |
Why Split at All?
Four independent reasons, all of which push toward smaller pieces:
| Reason | Explanation |
|---|---|
| Embedding limits | Embedding models accept a bounded input — commonly 512 to 8,192 tokens. Longer text is silently truncated by many libraries, meaning the tail of your document is embedded as nothing at all. |
| Precision | One vector must represent one chunk's meaning. A vector for a 40-page document represents everything and therefore nothing — it is an average, and averages match poorly. |
| Cost | You pay per token of context, per question, forever. Sending one paragraph instead of one chapter is a large, permanent saving. |
| Attention | Models attend unevenly across long contexts. Facts in the middle of a large block are measurably more likely to be overlooked. |
And one strong reason pushing the other way:
| Counter-pressure | Explanation |
|---|---|
| Context loss | A small chunk may not contain enough surrounding information to be understandable — or even interpretable — on its own. |
Chunking is the management of that tension.
See It Live
Drag the sliders. Watch what happens to the sentence explaining the drug interaction as chunks shrink.
Chunking visualizer
The same clinical note, split three ways. Watch what happens to the sentence explaining the drug interaction as you shrink the chunks.
Cuts at the nearest separator (paragraph, then sentence, then word) before the limit. The sensible default.
Chunks
5
Avg size
~29
tokens
Storage cost
121%
of original
The key fact survived
One chunk contains the full phrase "warfarin and ibuprofen", so a search for the drug interaction can retrieve it.
Patient was admitted on March 3 with shortness of breath. Chest imaging showed bilateral infiltrates. She was started on warfarin 5 mg daily for atrial fibrillation.
aily for atrial fibrillation. Her INR on admission was 2.4, within the target range of 2.0 to 3.0. On day two the team added ibuprofen for joint pain.
ded ibuprofen for joint pain. By day four the INR had risen to 4.8 and she developed bruising on both arms.
eloped bruising on both arms. The interaction between warfarin and ibuprofen was identified as the likely cause. Ibuprofen was stopped and vitamin K was given.
pped and vitamin K was given. The INR returned to 2.6 within 48 hours and she was discharged on day seven.
That red verdict is the whole lesson. When the key phrase splits across a boundary, no chunk answers the question any more. Retrieval is not the weak part here. The fact simply no longer exists as a findable unit anywhere in your index.
The Strategies
1. Fixed-Size Chunking
Cut every N characters or tokens. That is the entire algorithm.
| How | Slice at an exact count |
| Pros | Trivial, fast, perfectly predictable sizes |
| Cons | Slices words, sentences, and facts without mercy |
| Use when | Prototyping, or text with no structure at all (logs, transcripts) |
Fixed-size chunking is what most tutorials show first because it is easy to explain. It is rarely the right production choice. If you take one upgrade from this page, upgrade to recursive.
2. Recursive Character Chunking
Try to split on the most meaningful separator available. Attempt paragraph breaks first; if a piece is still too big, try single newlines; then sentences; then words; then raw characters as a last resort.
| How | Walk a separator list ["\n\n", "\n", ". ", " ", ""] from most to least meaningful |
| Pros | Respects natural boundaries; sizes stay near the target; simple and fast |
| Cons | Still blind to meaning — a topic shift mid-paragraph is invisible to it |
| Use when | Default choice. Start here unless you have a specific reason not to |
Recursive chunking is the right default for the overwhelming majority of systems. It captures most of the benefit of smarter strategies at a fraction of the complexity and none of the cost.
3. Document-Structure Chunking
Split on the document's own structure: Markdown headings, HTML sections, PDF bookmarks, legal clause numbers.
| How | Parse structure, split at headings, attach the heading path to each chunk |
| Pros | Boundaries fall exactly where the author intended topics to change; heading paths make excellent context prefixes |
| Cons | Requires reliable structure; section sizes vary wildly |
| Use when | Documentation, wikis, contracts, structured reports — anything with real headings |
Chunk text stored:
"[Anticoagulation Protocol 2025 > 4. Dose Adjustment > 4.2 Elderly Patients]
Reduce the starting dose by 50% in patients over 75 years of age."That bracketed path costs a handful of tokens and makes an otherwise ambiguous chunk both self-contained and far more retrievable.
4. Semantic Chunking
Embed each sentence, measure the similarity between consecutive sentences, and split where similarity drops — the point where the topic changed.
| How | Embed sentences → compute consecutive similarity → split at dips below a threshold |
| Pros | Boundaries follow actual meaning, not formatting |
| Cons | Expensive (an embedding call per sentence at index time); threshold needs tuning; produces highly variable sizes |
| Use when | Unstructured prose where topics shift without headings — transcripts, interviews, essays |
Semantic chunking sounds obviously superior and often measures only slightly better than recursive on real corpora, at several times the indexing cost. Measure it against your baseline before adopting it. This is a recurring theme: the sophisticated option is not automatically the better one.
5. Parent-Document Retrieval
Two sizes at once. Index small chunks so retrieval is precise; return the large parent chunk so the model gets full context.
Parent-document retrieval
| Pros | Escapes the size trade-off — precise matching and full context |
| Cons | Extra storage and a lookup table; returned context is larger, so it costs more tokens |
| Use when | Chunks are individually precise but too fragmentary to answer from |
This is one of the highest-value upgrades available and is under-used. If your symptom is "the right chunk is retrieved but the answer is incomplete," this is very likely your fix.
6. Contextual Chunk Headers
Prepend a short generated summary of the parent document to every chunk before embedding it.
Original chunk:
"Reduce the starting dose by 50%."
Contextualised chunk:
"From the 2025 anticoagulation protocol, section on elderly dosing:
Reduce the starting dose by 50%."The second version retrieves far better because the embedding now encodes what this is about, not merely the bare instruction. Cost: one small model call per chunk at index time, paid once.
7. Late Chunking
Embed the whole document first, so every token's representation is informed by the full document, then pool those token representations into chunk vectors afterwards.
| How | Long-context embedding model → token embeddings for the whole document → pool per chunk |
| Pros | Each chunk vector carries document-wide context; pronouns and references resolve naturally |
| Cons | Requires a long-context embedding model; more complex indexing |
| Use when | Documents with heavy cross-referencing where chunks reference each other constantly |
Strategy Comparison
| Strategy | Index cost | Quality | Complexity | Good default? |
|---|---|---|---|---|
| Fixed-size | Lowest | Poor | Trivial | Prototypes only |
| Recursive | Low | Good | Low | Yes — start here |
| Document-structure | Low | Very good | Low-medium | Yes, when structure exists |
| Semantic | High | Good-to-very-good | Medium | Only after measuring |
| Parent-document | Medium | Very good | Medium | Strong upgrade |
| Contextual headers | Medium-high | Very good | Medium | Strong upgrade |
| Late chunking | High | Very good | High | Specialised |
A sane adoption order
- Start with recursive, 400–800 tokens, 10–15% overlap.
- Add document-structure splitting if your documents have headings.
- Measure. If context is missing from answers, add parent-document retrieval.
- If chunks are ambiguous in isolation, add contextual headers.
- Consider semantic or late chunking only if measurement says the earlier steps were not enough.
Choosing Chunk Size
There is no universally correct size — but the trade-off is completely predictable.
The size trade-off
Small chunks — roughly 100-250 tokens
Sharper matching: the vector represents one idea, so similarity scores are meaningful. Cheaper context. But: facts split easily, chunks often lack the context needed to interpret them, and you need more of them to cover an answer.
Medium chunks — roughly 400-800 tokens
RecommendedThe practical sweet spot for most prose. Usually holds a complete idea with its surrounding context, still specific enough for precise matching. Start at 500 and adjust based on measurement.
Large chunks — 1,000+ tokens
Rich context, few boundary breaks. But the vector becomes an average of several topics and matches everything vaguely; token costs climb; and the model may overlook facts buried in the middle.
Size by Content Type
| Content | Suggested size | Reasoning |
|---|---|---|
| FAQ pairs | 100–300 tokens | One question and answer is naturally self-contained |
| Documentation | 400–800 tokens | One concept plus its explanation |
| Legal contracts | 500–1,500 tokens | Clauses are long and must stay whole; wording is exact |
| Research papers | 500–1,000 tokens | Paragraph or subsection level |
| Chat transcripts | 200–500 tokens | Turn or short exchange level |
| Code | Function or class | Never split mid-function — structure is the boundary |
| News articles | 300–600 tokens | Paragraph level |
Choosing Overlap
Overlap means each chunk repeats the last N tokens of the previous one. It is insurance against boundary damage: a fact sitting on a boundary appears whole in at least one chunk.
| Overlap | Effect |
|---|---|
| 0% | Cheapest. Any fact on a boundary is destroyed. Only safe with structure-aware splitting |
| 10–15% | Recommended default. Meaningful protection, modest storage cost |
| 20–25% | Strong protection. Noticeable duplication in results — deduplicate before assembling context |
| >30% | Rarely justified. Heavy storage cost, and near-duplicate chunks crowd your top-k results |
Overlap has a hidden cost beyond storage: duplicate content in retrieval results. Two overlapping chunks both matching a query occupy two of your top-k slots while conveying nearly the same text. Deduplicate at the context-assembly stage.
Chunking Special Content
| Content type | Rule |
|---|---|
| Tables | Keep whole if possible. If splitting, repeat the header row in every piece |
| Code | Split on function or class boundaries. A half-function is unusable |
| Lists | Keep the introducing sentence with the list — orphaned bullets lose all meaning |
| Q&A pairs | Never separate a question from its answer |
| Legal clauses | Split on clause numbers; keep any clause whole regardless of length |
| Dialogue | Keep exchanges together; a lone reply without its question is meaningless |
| Images and captions | Keep the caption with any surrounding description of the figure |
Concept Checks
Check yourself
Retrieval finds the right chunk, but answers keep missing detail. What do you change?
Your chunks are too small or too fragmentary — matching is working, context is not. The targeted fix is parent-document retrieval: keep indexing the small chunks so matching stays precise, but return the larger parent so the model has the surrounding detail. Simply increasing chunk size also works but costs you matching precision.
Why does 0% overlap sometimes cost nothing, and sometimes cost everything?
It depends on whether your boundaries are meaningful. With document-structure splitting, boundaries fall at headings — places the author already treated as topic breaks — so nothing important spans them. With fixed-size splitting, boundaries fall at arbitrary character counts and will eventually land mid-fact. Overlap is insurance priced against how arbitrary your boundaries are.
Semantic chunking measured only marginally better than recursive but costs 6× more to index. Adopt it?
No. That is the measurement telling you the simpler option is sufficient for your corpus, and you should spend the effort elsewhere — hybrid search or reranking will almost certainly return more. Reach for semantic chunking when the specific symptom is topic drift inside long unstructured prose, and re-measure then.
Why is a single vector for a 40-page document nearly useless?
Because an embedding is a fixed-length summary of meaning. Compressing forty pages spanning many topics into one point produces an average that sits near the centre of the space — moderately similar to many queries and strongly similar to none. It matches weakly and indiscriminately, which is the worst combination for ranking.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Chunking | Splitting documents into retrievable units — the highest-leverage RAG decision |
| Boundary damage | A fact split across chunks becomes unretrievable, permanently |
| Fixed-size | Simple, blunt; prototypes only |
| Recursive | Splits at the best available separator — the default |
| Document-structure | Splits at headings; gives free context prefixes |
| Semantic | Splits where meaning shifts; costly, measure before adopting |
| Parent-document | Index small for precision, return large for context |
| Contextual headers | Prepend document context so chunks stand alone |
| Size default | 400–800 tokens for most prose; start at 500 |
| Overlap default | 10–15%; deduplicate results afterwards |
Next
Your chunks are ready. Now turn them into numbers a computer can compare: Embeddings Explained.
Document Loading & Parsing
Turning PDFs, HTML, tables, and scans into clean text — the stage that silently caps the quality of everything downstream
Embeddings Explained
What a vector actually is, how cosine similarity works, how to choose an embedding model, and the mistakes that silently destroy retrieval