Training Data & Curricula
Why data quality matters more as models shrink, the multi-stage curriculum pattern, and why small models are trained past the compute-optimal point on purpose
Training Data & Curricula
TL;DR
A large model can partly absorb noisy or redundant training data because it has capacity to spare. A small model can't — every parameter is scarce, so what it trains on determines what it becomes far more directly. Modern small models are trained through a staged curriculum, not a single undifferentiated pass, and are usually trained on far more tokens than "compute-optimal" would suggest, because they're trained once but served enormously many times.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~18 minutes |
| Prerequisites | SLM Architectures |
| You will understand | Why small-model training curricula are staged, and the single most important cost trade-off behind how they're sized |
Capacity Is Scarce, So Data Quality Isn't Optional
A frontier model trained on a huge, somewhat noisy web corpus can still turn out excellent, because it has enough parameters to learn the useful signal and mostly ignore the noise. A 1–4B model doesn't have that luxury — every parameter spent modeling noise is a parameter not spent on something useful.
This is why small-model training has moved toward "less but better" rather than "more." Aggressive filtering, deduplication, and a preference for curated or synthetic data over raw scraped text aren't nice-to-haves for small models — they're close to the whole game. A small model trained on twice the (unfiltered) data is not reliably better than one trained on half as much, carefully filtered.
The Multi-Stage Curriculum
Leading small models aren't trained in one undifferentiated pass over a giant corpus. They move through distinct stages, each targeting a different capability:
A typical small-model training curriculum
Why code and math upweighting helps general tasks
This surprises people the first time they see it: increasing the proportion of code and mathematical reasoning in pretraining measurably improves a model's performance on tasks that have nothing to do with code or math — following multi-step instructions, reasoning through a word problem, structuring an answer logically. The leading explanation is that code and math are unusually dense in explicit, checkable logical structure, and a model that's absorbed more of that structure generalizes it to reasoning tasks in general. At small scale, where every training signal counts more, this upweighting is standard practice rather than an optional tweak.
Synthetic data as a standard tool, not a shortcut
Using a larger, stronger model to generate or filter training examples for a smaller model is now a default part of the pipeline, not a workaround. It shows up in two forms:
| Form | What it does |
|---|---|
| Generation | A strong teacher model writes example instructions, answers, or reasoning traces that become training data |
| Filtering | A strong model scores or filters a large raw corpus, keeping only the examples that meet a quality bar a human reviewer would set |
Both feed directly into Knowledge Distillation, which goes further — training the student to imitate the teacher's behavior directly, rather than just training on teacher-curated data.
The Most Important Idea on This Page: Overtraining on Purpose
There's a well-known result (often called "Chinchilla-optimal") describing the ratio of training tokens to parameters that minimizes training compute for a given final training loss. It's a genuinely useful result — but it answers a narrower question than people often assume.
Compute-optimal for training is not optimal for serving
The Chinchilla-style calculation minimizes the cost of the training run. It says nothing about the cost of running the resulting model afterward. A model is trained once. If it's going to be served millions or billions of times, the cost that actually matters in the long run is inference cost, not training cost — and those two costs are minimized by different things.
Here's the reasoning that follows from that distinction:
Why small models are trained past the 'optimal' point
Start from the task
You need a model that fits a fixed deployment budget — a certain memory footprint, a certain latency
Parameter count is now fixed by the deployment target
Not by what's cheapest to train
Train on more tokens than 'optimal' for that parameter count
Costs more compute up front, but improves the fixed-size model's quality
Pay that extra training cost once
Recoup it across every one of the millions of inference calls the smaller model makes possible
This is the standard explanation for why current small models — the Llama-3.2 1B/3B pair, SmolLM3, the smaller Qwen3 variants — are trained on far more tokens than a naive compute-optimal calculation for their parameter count would suggest. The extra training compute buys a smaller, cheaper-to-serve final model, and at serving scale that trade wins decisively.
Two different optimization targets
Minimize training compute
The classical compute-optimal ratio. Sensible when the model will be trained and evaluated once, or when training compute genuinely is the binding constraint.
Minimize total cost including inference at scale
RecommendedThe right target for almost every deployed small model. Spend more at training time to shrink the parameter count (or hold it fixed and improve quality within it) — because inference happens far more often than training does.
Deduplication and Filtering: Unglamorous, High-Leverage
Two steps rarely get top billing in a model's release blog post, but do a disproportionate amount of the actual quality work:
| Step | What it fixes |
|---|---|
| Deduplication | Removes near-identical passages that would otherwise be seen (and over-weighted) many times, wasting training signal and increasing the risk of verbatim memorization |
| Quality filtering | Removes low-value content (boilerplate, spam, broken text extraction) before it ever competes for a small model's limited capacity |
For a large model, skipping aggressive deduplication mostly wastes some compute. For a small model, it wastes a much larger fraction of a much smaller total budget — which is why small-model training pipelines tend to invest more, not less, in this unglamorous cleanup stage.
Concept Checks
Check yourself
Why does data quality matter more for a 1B model than a 70B model, given the same underlying noisy dataset?
A larger model has spare parameter capacity to partially absorb or average out noisy, low-value, or redundant examples while still learning the useful signal elsewhere. A small model has far less spare capacity — every parameter spent modeling noise is a parameter not available for something useful, so noise in the data translates more directly into a worse model.
A team trains a 2B model on the compute-optimal (Chinchilla) token count for that size and is surprised a competitor's 2B model, trained on 5x more tokens, performs better despite costing more to train. What happened?
The compute-optimal ratio minimizes training compute for a given loss — it doesn't account for the fact that the model will be served many more times than it's trained. The competitor deliberately overtrained relative to that ratio, paying more upfront in training compute to get a better model at a fixed, deployment-constrained parameter count — a trade that pays off across enough inference calls.
Why does upweighting code and math in pretraining data improve performance on tasks unrelated to code or math?
Code and mathematical content is unusually dense in explicit, checkable logical structure. A model trained with more exposure to that structure appears to generalize the underlying reasoning patterns to other tasks — following multi-step instructions, structuring an answer logically — even when those tasks have nothing to do with code or math directly.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Capacity scarcity | Small models can't absorb noisy data the way large models can — quality matters more |
| Multi-stage curriculum | Web pretraining → code/math upweighting → synthetic data → instruction tuning → alignment |
| Code/math upweighting | Improves general reasoning, not just code/math performance |
| Synthetic data | A larger model generates or filters training examples for the smaller student |
| Chinchilla-optimal | Minimizes training compute for a given loss — says nothing about inference cost |
| Overtraining on purpose | Spending more training compute than "optimal" to get a smaller, cheaper-to-serve model |
| Deduplication/filtering | Unglamorous steps that matter proportionally more at small scale |
Next
Curricula shape what a model learns from data directly. The next step compresses what a larger model already knows into a small one: Knowledge Distillation.
SLM Architectures
The architectural techniques — GQA, tied embeddings, sparse MoE, and long-context tricks — that let small models punch above their parameter count
Knowledge Distillation
Teacher-student training — black-box vs logit-matching, why on-policy distillation matters, and TRL's GKDTrainer/DistillationTrainer/MiniLLMTrainer