Evaluation for SLMs
Why general knowledge benchmarks are the wrong primary signal for a deployed small model, and what to measure instead
Evaluation for SLMs
TL;DR
General knowledge benchmarks like MMLU reward breadth — exactly what a small model deliberately trades away on purpose. Evaluate what you actually shipped it to do: task accuracy on your own held-out set, instruction-following on multi-step prompts, tool-call validity, and — the step most teams skip — the quantized model you're actually deploying, not the fp16 checkpoint it was trained as.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~16 minutes |
| Prerequisites | SLM RAG |
| You will understand | What to measure for a deployed small model, and why a leaderboard score won't tell you |
The Leaderboard Trap, Sharper for Small Models
A general knowledge benchmark score measures how much of a broad distribution of facts and reasoning patterns a model absorbed during training. A small model, by design, absorbed less of that breadth than a large one — that's not a flaw, it's the trade being made deliberately for latency, memory, and cost. Scoring it primarily on a benchmark built to reward breadth is measuring the thing it was never optimized for, and it will look worse than it is at the job it was actually built to do.
A small model fine-tuned tightly on one task can lose ground on general benchmarks while getting better at the job you shipped it for. That's expected, not a regression — narrowing what a model is good at is often the entire point of building an SLM in the first place, per What Is a Small Language Model?. Don't let a dropping MMLU score talk you out of a fine-tune that's measurably improving the metric that matters.
What to Measure Instead
| Check | Why it matters specifically for SLMs |
|---|---|
| Task accuracy on your own held-out set | The only number that reflects the narrow job the model was actually built and fine-tuned for |
| Instruction-following on multi-step prompts | The most common place small models fall visibly behind large ones — test this deliberately, not just single-turn accuracy |
| Tool-call validity rate | Track this as its own metric, separate from overall task success — a malformed or wrong tool call is a distinct failure mode covered in SLM Agents & Tool Use |
| Latency and memory on the real target device | A benchmark score is meaningless if the model doesn't fit in the phone's available RAM, or takes twelve seconds per response on the actual hardware |
| Quantized vs fp16 regression | Score the artifact you're actually shipping — see below, this is the step most commonly skipped |
The Step Most Teams Skip: Re-Score After Quantizing
Training produces an fp16 (or bf16) checkpoint. Deployment usually ships something quantized down from it — GGUF Q4_K_M, Q5_K_M, or another format from Quantization for Edge. Quantization can introduce a real accuracy regression, and that regression is invisible unless you actually re-run evaluation on the quantized artifact.
eval_results.json (never actually generated by most teams):
fp16 baseline task_accuracy: 0.91 | model size: 2.4 GB
Q8_0 task_accuracy: 0.91 | model size: 1.3 GB ← no measurable loss
Q5_K_M task_accuracy: 0.89 | model size: 0.9 GB ← small, worth it for the size
Q4_K_M task_accuracy: 0.84 | model size: 0.7 GB ← real drop — is it acceptable?"We evaluated the model" and "we evaluated the model we shipped" are different claims. The gap between them is exactly where quantization regressions hide — a model that scored 91% in the training notebook can ship at a meaningfully lower real-world accuracy if nobody re-ran the eval on the quantized file that actually reaches users.
This is also the table that turns "which quantization level should we ship" from a guess into a decision — pick the most aggressive quantization level whose accuracy drop is still acceptable for the task, rather than defaulting to whatever level happened to fit a size target.
Measuring on the Real Device
A model that runs acceptably fast on a development machine can be unusable on the actual target hardware — a mid-tier phone, an older laptop, a browser tab competing for the same GPU as everything else open. Two numbers matter beyond accuracy:
| Metric | What it catches |
|---|---|
| Time to first token | Whether the app feels responsive the moment a user asks something |
| Peak memory during generation | Whether the app gets killed by the OS under memory pressure — this is often worse than slow, since it fails silently |
Measure these on the lowest-spec device you intend to support, not the newest one on the team's desk. A model that's comfortable on a flagship phone can be genuinely unusable on a three-year-old mid-range one, and that's exactly the device population most likely to be under-served if nobody tests it.
Building a Held-Out Set That Actually Tells You Something
The same discipline from Evaluation applies here, with one SLM-specific addition: your held-out set should include the multi-step and tool-use cases where small models are known to struggle, not just easy single-turn examples that any model size would pass. A test set made entirely of easy cases will make a poorly-tuned small model look fine right up until it meets a real multi-step request in production.
Concept Checks
Check yourself
A fine-tuned small model's MMLU score drops after fine-tuning, while its accuracy on the target task goes up. Is this a problem?
Not necessarily — and often it's the expected outcome. General knowledge benchmarks reward breadth, which is exactly what narrowing a model toward one task trades away on purpose. The metric that should drive the decision is task accuracy on your own held-out set, not a general benchmark the model was never optimized to maximize.
Why is it not enough to evaluate the fp16 training checkpoint before shipping?
Because deployment usually ships a quantized artifact — GGUF Q4_K_M or similar — derived from that checkpoint, and quantization can introduce a real accuracy regression that's invisible unless the quantized model itself is re-evaluated. Scoring only the fp16 checkpoint measures a model nobody actually runs in production.
Why measure latency and memory on the lowest-spec supported device rather than a development machine?
Because performance and memory headroom on a fast development machine or a flagship phone don't represent the experience on the actual device population, especially older or mid-range hardware most likely to be memory-constrained. A model that's comfortable on the team's own hardware can be unusable — or get killed by the OS under memory pressure — on the devices real users actually have.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| The leaderboard trap | General benchmarks reward breadth, which a small model deliberately trades away |
| Task accuracy on held-out data | The number that actually reflects the job the model was built for |
| Tool-call validity rate | Tracked separately from overall accuracy, not folded into it |
| Re-score the quantized artifact | Evaluate what's actually shipped, not the fp16 training checkpoint |
| Device-realistic measurement | Latency and memory on the lowest-spec supported device, not a dev machine |
| A dropping general score isn't always bad | Expected when a model is deliberately specialized for a narrow task |
Next
With a trustworthy way to measure the system, the next page covers running it in production: Production & Operations.
SLM RAG
Why retrieval and small models pair naturally, how to build a fully local RAG stack, and the tighter context budget a small model needs
Production & Operations
Picking model size from the task, the real cost math, the escalation-routing pattern, and treating on-device model updates like deploys