Speculative decoding is a trick for making large language models generate text faster without changing what they say. A small, fast draft model guesses several tokens ahead, and the large target model checks all those guesses in a single parallel pass. Correct guesses are kept for free; wrong ones are thrown away and corrected. The output is identical to what the big model would have produced alone — you just reach it in fewer slow steps.
What changed in 2026
- It became a default, not an experiment. Most serving stacks now ship speculative decoding as a built-in option rather than a research feature you bolt on.
- Self-speculation matured. Techniques that let a model draft using its own early layers or extra prediction heads — removing the need for a separate draft model — became practical and widely used.
- Draft-target tuning got easier. Off-the-shelf recipes for pairing and calibrating models raised acceptance rates, which is where real-world speedups actually come from.
Why generation is slow to begin with
Standard autoregressive decoding produces one token at a time. Each new token requires a full forward pass through the model, and the next token cannot start until the current one finishes. On modern hardware this is usually limited by memory bandwidth, not raw compute, so the accelerator sits partly idle waiting to move weights. Speculative decoding fills that idle time by verifying several candidate tokens in one pass.
How speculative decoding works
- The draft model proposes a short run of tokens, say four, by generating them quickly.
- The target model runs a single forward pass over the prompt plus all four proposed tokens, producing its own probability for each position.
- Using rejection sampling, the system accepts the longest prefix of the draft that is consistent with the target distribution.
- At the first token the target would not have produced, it substitutes its own token and the loop repeats.
The key property: because acceptance is governed by the target model probabilities, the final text is statistically identical to plain decoding with the target model. You are trading a little extra compute for far fewer sequential steps.
What determines the speedup
| Factor |
Effect on speedup |
| Acceptance rate |
Higher acceptance means more free tokens per pass |
| Draft model cost |
A cheaper draft leaves more of the savings intact |
| Number of tokens drafted (k) |
Larger k helps if acceptance stays high, hurts if it drops |
| Domain match |
A draft aligned to your data accepts far more often |
| Batch size |
Gains shrink as batching already saturates the hardware |
Approaches compared
- Separate draft model. A small model of the same family drafts for a large one. Simple and effective, but you host two models.
- Self-speculative (extra heads or early exit). The target model drafts for itself, avoiding a second model and its memory cost.
- Prompt lookup and n-gram drafting. Cheap heuristics that copy likely continuations from the prompt; strong for repetitive or retrieval-heavy text, weaker for open-ended generation.
Speculative decoding pairs naturally with other efficiency techniques. If you are also shrinking model size, see our guide to quantization explained in 2026.
When it helps and when it does not
It shines for latency-sensitive, single-stream generation — chat, coding assistants, agents — where the hardware would otherwise idle between tokens. It helps less when you are already running very large batches that keep the accelerator busy, because there is little slack to reclaim. And a poorly aligned draft can even slow things down, since rejected tokens mean wasted verification work.
Pitfalls to watch
- A mismatched draft. If the draft rarely agrees with the target, acceptance craters and you lose the benefit.
- Extra memory. A separate draft model competes for the same accelerator memory as the target.
- Over-tuning k. Drafting too many tokens ahead wastes work when acceptance falls off toward the end of the run.
FAQ
Does speculative decoding change the model output?
No. Correctly implemented, it produces the same distribution as decoding with the target model alone.
How much faster is it?
It depends entirely on acceptance rate and draft cost, so verify on your own workload. Well-matched setups commonly see meaningful multiples on single-stream latency.
Do I need a separate draft model?
Not anymore. Self-speculative methods let a model draft for itself, which avoids the memory and hosting cost of a second model.
Is it the same as parallel decoding?
It is one family of parallel-verification methods. The defining idea is guess-then-verify against the target distribution.
Where to go next