Large language models generate text one token at a time, and each token requires a full pass through the model. For a big model, that pass is slow, not mainly because of arithmetic but because the hardware must read billions of weights from memory for every single token. The GPU spends much of its time waiting.
Speculative decoding exploits that idle capacity. A small, fast model guesses several tokens ahead; the large model checks all the guesses in one pass, keeping the ones it agrees with. When the guesses are good, you get several tokens for roughly the cost of one large-model step. And with the right acceptance rule, the output follows exactly the same probability distribution as the large model alone. This explainer covers how it works, why it is lossless, what determines the speed-up, popular variants and practical limits.
Why generation is slow
Generating a token with a large model involves reading all of its weights from GPU memory. For one sequence at a time, the amount of computation per byte read is low, so throughput is limited by memory bandwidth, not compute. The same reasoning explains why local models run faster when quantized; see how to run an 8B LLM on a laptop.
The key observation: checking several tokens at once costs about the same as generating one, because the weights only need to be read once for the whole batch of positions. If you already had good candidate tokens, the large model could verify them in parallel.
How speculative decoding works
Two papers introduced the technique independently: Leviathan, Kalman and Matias (2023), "Fast Inference from Transformers via Speculative Decoding," and Chen et al. (2023), "Accelerating Large Language Model Decoding with Speculative Sampling."
The loop:
- Draft. A small draft model generates *k* tokens, for example 4, one after another. This is fast because the draft model is small.
- Verify. The large target model processes the prompt plus all *k* draft tokens in a single forward pass, producing its own probabilities for each position.
- Accept or reject. Starting from the first draft token, each is accepted with a probability based on how the target model's probability compares with the draft model's. The first rejected token is replaced by a sample from an adjusted distribution, and the rest of the draft is discarded.
- Bonus token. If all *k* drafts are accepted, the target model's pass also yields one more token for free.
- Repeat from the new position.
Prompt: "The capital of France is"
Draft (small model): " Paris" "." " It" " is"
Target verifies in one pass:
" Paris" accept
"." accept
" It" reject → target samples " The" instead
Result this step: " Paris" "." " The" (3 tokens for 1 target pass + 4 cheap draft steps)Why the output does not change
The acceptance rule is a form of rejection sampling. Both papers proved that, with this rule, the final tokens follow exactly the target model's distribution, the same as if the target model had generated every token itself. With greedy decoding, the output is identical token for token (up to floating-point effects).
This is what distinguishes speculative decoding from approaches that trade quality for speed, such as quantization or using a smaller model outright. The large model's outputs stay the same; only the time to produce them changes.
What determines the speed-up
- Acceptance rate: how often the target model agrees with the draft. Predictable text (code, boilerplate, structured output, common phrases) gives high acceptance; creative or unusual text gives lower acceptance.
- Draft cost: the draft model must be much faster than the target. A draft that is too large eats the savings.
- Draft length *k*: longer drafts give more potential tokens per step but are more likely to be cut short by a rejection.
- Batch size: at large batch sizes on busy servers, GPUs become more compute-bound, and the spare capacity speculative decoding exploits shrinks.
The original papers reported speed-ups of roughly 2× to 3× in their settings without changes to outputs: Leviathan et al. on T5-XXL and Chen et al. on the 70B Chinchilla model, which they reported at roughly 2 to 2.5×. Real-world gains vary with the workload.
Variants
| Variant | Where the draft comes from | Trade-off |
|---|---|---|
| Classic draft model | A separate smaller model from the same family | Needs a compatible tokenizer and a good small model |
| Medusa (Cai et al., 2024) | Extra prediction heads on the target model | No separate model; heads must be trained |
| EAGLE (Li et al., 2024) | A lightweight head predicting from target features | Often high acceptance; needs training |
| Prompt lookup / n-gram drafts | Copying spans from the prompt or earlier output | Free drafts; great for editing and summarising |
| Self-speculative | Skipping layers of the target model to draft | No extra model; acceptance varies |
Prompt-lookup decoding is worth highlighting for builders: when the output repeats large parts of the input, as in code edits, document rewriting or extraction, simply proposing tokens copied from the prompt can give large speed-ups with no extra model.
Where you will see it
- Inference servers for open models increasingly support speculative decoding through configuration, with a draft model or built-in heads.
- Hosted APIs may use it internally; you see the effect as faster token rates, especially on predictable outputs.
- Local tools such as llama.cpp support draft models, which can noticeably speed up large models on consumer hardware.
Choosing a draft model
If you run open models yourself and use the classic approach, the draft model choice decides most of the gain:
- Same family, much smaller. A 1B to 3B model from the same family as a 70B target is a typical pairing. Shared training data and tokenizer lead to higher acceptance.
- Aim for a large speed ratio. The draft should run many times faster than the target; otherwise its own cost cancels the benefit.
- Fine-tuned targets need matching drafts. If the target model is fine-tuned for a domain or format, a draft fine-tuned on similar data accepts far more often.
- Measure acceptance rate directly. Most inference servers report it. Tune the draft length *k* together with the draft model: high acceptance supports longer drafts.
Limits and practical advice
- Measure on your workload. Gains depend on how predictable your outputs are. Use the methods in designing an LLM latency benchmark.
- Tokenizer compatibility matters for classic draft models: the draft and target should share a vocabulary.
- Memory overhead: a draft model takes extra GPU memory, which can reduce how many requests fit at once.
- Throughput vs. latency: speculative decoding mainly improves latency for individual requests; for maximum throughput under heavy load, batching may matter more.
It combines with other efficiency techniques: quantization shrinks the weights that must be read, mixture of experts reduces computation per token, and sparse attention lowers the cost of long contexts.
Frequently asked questions
What is speculative decoding?
Speculative decoding is an inference technique in which a fast draft model proposes several tokens and the large target model verifies them in a single pass, accepting those it agrees with. It reduces latency while preserving the target model's output distribution.
Does speculative decoding change the model's output?
No. With the standard acceptance rule, outputs follow exactly the same distribution as the target model alone; with greedy decoding they are identical apart from floating-point effects.
How much faster is speculative decoding?
Published results report around 2 to 3 times faster generation in favourable settings. Actual gains depend on how often the draft is accepted, how cheap the draft is, and how busy the hardware is.
What is a draft model?
A draft model is a smaller, faster model, usually from the same family and sharing the same tokenizer, that proposes candidate tokens for the large model to verify.
