Speculative Decoding
How a small draft model makes a large target model faster — the mechanism, why it works, and the alternatives that don't need a second model at all
Speculative Decoding
TL;DR
A small "draft" model proposes several tokens ahead; a large "target" model verifies all of them in a single forward pass instead of generating one token at a time. Because LLM decoding is usually memory-bandwidth-bound rather than compute-bound, verifying several tokens costs barely more than verifying one — so correct guesses are nearly free, and the result is typically a 2-3x wall-clock speedup with no change to output quality.
| Property | Value |
|---|---|
| Level | Intermediate |
| Reading time | ~16 minutes |
| Prerequisites | On-Device & Edge Deployment |
| You will understand | How speculative decoding works, why it's a genuine speedup and not an approximation, and where small models fit into speeding up large ones |
The Idea No One Expects
Everything else in this track is about running a small model instead of a large one. This page is different: it's about using a small model alongside a large one, purely to make the large model faster — with no change to the large model's output.
One round of speculative decoding
The output is identical to what the target model would have generated token-by-token on its own — this isn't an approximation or a quality trade-off, it's a way of getting the exact same result faster.
Why This Actually Works
The core fact that makes this work: LLM decoding is usually memory-bandwidth-bound, not compute-bound. Generating one token means streaming the entire model's weights through the GPU once — the arithmetic itself is comparatively cheap. Verifying five draft tokens in one pass touches the same weights once, not five times, so it costs only marginally more than verifying one.
That asymmetry is the whole mechanism. If drafting were as expensive as running the target model, there'd be no win. Because a small draft model is cheap to run repeatedly and the target model's verification pass is nearly as cheap as its single-token generation pass, the net effect is more tokens produced per expensive target-model pass — without ever letting the draft model's own (lower) quality reach the output, since every token is checked.
Using It in transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
target = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", dtype="bfloat16")
draft = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
inputs = tokenizer("Explain speculative decoding in one paragraph.", return_tensors="pt")
outputs = target.generate(**inputs, assistant_model=draft, max_new_tokens=200)assistant_model= is the entire API surface — the rest is handled internally. The draft model is conventionally a smaller model from the same family, since it tends to agree with the target model more often on easy, predictable tokens.
When the draft and target don't share a tokenizer
Normally the draft model needs the same tokenizer as the target, since verification compares token IDs directly. Universal assisted generation removes that restriction — GenerationConfig's assistant_lookbehind and target_lookbehind parameters re-encode a window of recent tokens between the two models' vocabularies, so the draft model doesn't have to be a smaller checkpoint of the exact same family. This matters in practice: it means an existing small model you already trust can serve as a draft for a target model it was never designed alongside.
Alternatives That Skip the Second Model
Text Generation Inference (TGI) documents two other speculation strategies that don't require training or hosting a separate draft model:
Speculation without a dedicated draft model
Medusa
Adds extra prediction heads directly onto the target model, each guessing a few tokens ahead. No second model to load or maintain, at the cost of the target model needing to be trained (or fine-tuned) with these extra heads in the first place.
N-gram speculation
Speculates using simple n-gram matches against the prompt and generation so far — no model involved at all. Cheapest option, and surprisingly effective for tasks with a lot of repeated structure (e.g. code, where identifiers and boilerplate recur constantly), weaker on genuinely novel text.
Pick based on what you're optimizing for: a separate small draft model is the most broadly effective option and the one covered above, Medusa avoids hosting a second model but needs the target model itself modified, and n-gram speculation needs nothing extra at all but works best where the output has repetitive structure.
Where This Fits in the SLM Track
This page is a deliberate detour from the rest of the track's throughline. Everywhere else, a small model is the thing being deployed. Here, it's a component that makes something else faster — a small model earns its place even in a system whose actual output comes from a large model. The two use cases aren't mutually exclusive either: a small model deployed standalone for simple queries and the same (or a different) small model serving as a draft for a larger model handling complex ones can coexist in one system, which is exactly the routing pattern covered in SLM Agents & Tool Use and Production & Operations.
Concept Checks
Check yourself
Why doesn't speculative decoding trade output quality for speed, the way quantization does?
Because every draft token is checked against the target model before being accepted — the final output is exactly what the target model would have generated one token at a time on its own. Quantization changes the model's actual computation; speculative decoding changes only how many forward passes it takes to reach the same result.
Why does verifying five draft tokens in one target-model pass cost barely more than verifying one?
Because LLM decoding is typically memory-bandwidth-bound: the dominant cost of a forward pass is streaming the model's weights through the GPU once, not the arithmetic on the input tokens. Verifying more tokens in that same pass adds a small amount of extra compute but doesn't multiply the expensive part — the weight transfer — so the marginal cost of checking more tokens at once is small.
What does universal assisted generation let you do that ordinary speculative decoding can't?
It lets the draft and target models use different tokenizers, by re-encoding a lookbehind window of recent tokens between the two vocabularies instead of requiring token-ID-level agreement. Ordinary speculative decoding requires a shared tokenizer, which usually restricts the draft model to a smaller checkpoint from the exact same model family.
Key Concepts Recap
| Concept | One-line summary |
|---|---|
| Draft model | A small, fast model that proposes tokens ahead of the large model |
| Target model | The large model whose exact output is being sped up, not approximated |
| Why it works | Decoding is memory-bandwidth-bound; verifying several tokens costs about the same as verifying one |
assistant_model= | The transformers API surface for speculative decoding |
| Universal assisted generation | Lets draft and target use different tokenizers via lookbehind re-encoding |
| Medusa | Extra prediction heads on the target model itself — no separate draft model |
| N-gram speculation | Pattern-matches against existing text — no model at all |
| No quality trade-off | Output is identical to standard target-model decoding, just faster |
Next
Speeding up a large model is one job for a small model. The next page covers making a small model reliable when it's the one actually deciding what to do: SLM Agents & Tool Use.
On-Device & Edge Deployment
The full stack from quantized model file to a shipped offline app, and the constraints — RAM, battery, download size, versioning — unique to running on someone else's device
SLM Agents & Tool Use
Why tool use is where the capability gap shows up most, and the three techniques that make a small model a reliable agent anyway