Most large language models work autoregressively, meaning that they generate every new token by predicting it from every token that came before it. Speculative decoding is a technique introduced in 2022–2023 by researchers at Google DeepMind that essentially makes it possible to predict multiple tokens in advance, which significantly speeds up generation on the same hardware.
In this article, then, we’ll explain what speculative decoding is, what impact it has on performance, and how to use it in Atomic Chat.
TLDR
Speculative decoding speeds up text generation without changing the model’s final output. Normally, a model generates every token individually. With speculative decoding, however, a model uses a lightweight “draft” step to predict several tokens in advance, then uses the larger “target” model to verify those predictions. Predictions that match are accepted, and the first incorrect prediction and everything after it is discarded. Since modern GPUs can often process a batch of tokens at nearly the same cost as a single token, accurate draft predictions significantly improve throughput with little extra computation. Atomic Chat ships its own tuned inference engine with speculative decoding built in — including Multi-Token Prediction (MTP) and DFLASH — so several of the methods described here are features you get out of the box rather than settings you have to configure.
What is speculative decoding?
Language models generate text autoregressively. This means that each new token is predicted from every token that came before it. During standard decoding, this means the model performs one forward pass for every token it produces. On large models, this process is often limited by memory bandwidth, because the GPU spends much of its time reading model weights from memory for each forward pass, while generating only a single new token. Speculative decoding reduces the number of these forward passes.
With speculative decoding, the model first generates a short sequence of candidate tokens using a cheap drafting mechanism, usually one of these three:
- A much smaller LLM called a draft model.
- Prediction heads which are attached to the target model and estimate several future tokens.
- Or by reusing previously seen text, such as repeated n-grams, to propose likely continuations.
Those candidates are then checked by the full model in a single verification pass.
Regardless of how the candidates are produced, the target model remains the only model that determines the final output. And because the verification step uses the same conditional probabilities the target model would have computed during ordinary autoregressive decoding, speculative decoding is completely lossless.
How speculative decoding works
Once a draft has been proposed, the target model evaluates it to determine which drafted tokens can be retained without changing the probability distribution that ordinary autoregressive decoding would have produced.
This is accomplished using a rejection-sampling algorithm. The target model walks through the drafted sequence from left to right, comparing each proposed token against its own conditional distribution. For a drafted token x, let p(x) be the probability the target model assigns it and q(x) the probability the drafter assigned it. The token is accepted with probability:
p_accept(x) = min(1, p(x) / q(x))
When the target is at least as confident as the drafter (p(x) ≥ q(x)), the token is always accepted. When the target is less confident, it is accepted in proportion to how much the two distributions agree.
- If the drafted token is accepted, verification continues with the next token.
- At the first rejected token, verification stops. That token is resampled from the residual distribution
norm(max(0, p(x) − q(x))), and every drafted token after it is discarded because it was conditioned on a prefix that is no longer valid.
This acceptance rule keeps the procedure exact: averaged over the drafter’s proposals, the surviving tokens follow the target model’s distribution p, so the output matches standard decoding.
Only the longest accepted prefix survives. Generation then resumes from the corrected sequence, beginning a new speculative iteration.
The effectiveness of speculative decoding is therefore determined primarily by the acceptance rate — the proportion of drafted tokens that survive verification.
- Higher acceptance rates allow each expensive target-model forward pass to advance the sequence by several tokens instead of one.
- Lower acceptance rates reduce the benefit because more drafted work is discarded.
This relationship can be made precise. If each drafted token is accepted with probability α (the acceptance rate) and the drafter proposes up to γ tokens per step, the expected number of tokens produced per verification pass is (Leviathan et al. (2023)):
E[tokens per step] = (1 − α^(γ+1)) / (1 − α)
This expression describes the expected number of tokens produced by a single target-model verification step before accounting for the computational cost of generating the draft.
For a draft length of γ = 4 and an acceptance probability of α = 0.7, the expected number of tokens produced per verification step is approximately 2.8 tokens. This represents a potential throughput increase of roughly 2.8× compared with standard decoding.
As the acceptance probability approaches 1, nearly the entire draft is accepted and the expected yield approaches the maximum possible value of γ + 1 tokens per verification step.
In practice the realized speedup is lower, because producing the draft is not free. Let c be the cost of one drafting step relative to one target-model forward pass. The wall-clock speedup over standard decoding is:
speedup = (1 − α^(γ+1)) / ((1 − α) × (γc + 1))
The numerator is the expected tokens per step from above; the denominator adds the drafting overhead γc to the single verification pass. A cheaper drafter therefore competes with an accurate one: a method with lower α but near-zero c, such as an n-gram lookup, can beat a more accurate neural drafter whose c is large.
With that in mind, there are multiple speculative decoding methods that balance the acceptance rate in different ways, and we break them down in the next section.
Types of speculative decoding methods
There are multiple speculative decoding methods that vary in how draft tokens are generated. Broadly speaking, there are two categories:
- Using an auxiliary model to approximate the target model’s predictions
- Directly from patterns in the existing context
The first category consists of model-based drafters that use a second neural network to propose candidate tokens for the target model to verify.
- Draft models use a separate, smaller language model trained to imitate the target model.
- EAGLE-3 attaches a lightweight prediction module that operates on the target model’s hidden activations.
- DFLASH uses a block-diffusion model that predicts an entire block of tokens in a single forward pass.
- Multi-Token Prediction (MTP) adds auxiliary prediction heads to the target model itself, proposing several tokens per forward pass.
The second category consists of pattern-based, or model-free, methods that exploit repetition already present in the generated text. Since they perform little or no additional computation, they introduce almost no inference overhead, although their effectiveness depends heavily on the prompt and the amount of repeated structure.
- N-gram cache constructs draft probabilities from previously observed n-gram statistics.
- N-gram simple searches the existing context for the most recent matching n-gram and proposes the tokens that followed the previous occurrence.
- N-gram map accelerates these lookups by maintaining a hash map over n-grams in the current context window.
- N-gram mod uses a fixed-size shared hash table to bound memory usage while preserving constant-time lookups.
These approaches are not mutually exclusive. In implementations such as llama.cpp, pattern-based methods can be attempted first because they incur almost no computational cost. If no suitable continuation is found, the decoder can fall back to a neural drafter, allowing inexpensive pattern matching and model-based speculation to complement one another.
Model support
The following EAGLE-3 speculators are supported:
| Target family | EAGLE-3 speculators |
|---|---|
| LLaMA 3.1 / 3.3 | EAGLE3-LLaMA3.1-Instruct-8B, EAGLE3-LLaMA3.3-Instruct-70B |
| Qwen3 | Qwen3-1.7B/4B/8B/14B/32B_eagle3, qwen3_8b_eagle3, qwen3_30b_moe_eagle3 |
| Gemma 4 | gemma-4-31B-it-speculator.eagle3, gemma-4-26B-A4B-it-speculator.eagle3 |
| GPT-OSS | gpt-oss-20b-speculator.eagle3, EAGLE3-gpt-oss-120b-bf16, gpt-oss-120b-Eagle3-long-context |
DFLASH support is provided through target-specific speculator models. For example, a Qwen3 target may use a corresponding DFLASH speculator such as Qwen3-4B-DFlash.
For Multi-Token Prediction (MTP), no separate model download or loading step is required. Only models that were trained with MTP support can use this method.
Speculative decoding in Atomic Chat
Atomic Chat is a local LLM app that allows you to run models from Hugging Face in a single click. It features a custom tuned inference engine with integrated speculative decoding.
Atomic Chat supports models in the GGUF, MLX, and ONNX formats, and exposes generation through an OpenAI-compatible local server at http://localhost:1337/v1. On supported models it applies speculative decoding automatically, using Multi-Token Prediction (MTP) and DFLASH.
Multi-Token Prediction (MTP) in Atomic Chat
In Atomic Chat, MTP provides a throughput improvement of 30–70% on supported models, with higher gains on models where the additional prediction heads maintain strong agreement with the target distribution. On Gemma 4, gains can reach approximately 3× under favorable workloads. For example, Qwen 3.6 27B running on two RTX 5090 GPUs increases from 51 to 117 tokens per second, representing roughly a 2.3× improvement.
In Multi-Token Prediction (MTP), during training, the model is augmented with additional prediction heads that learn to predict tokens at future positions in the sequence. Each head corresponds to a different offset from the current token position.
During inference, the model’s normal output head predicts the next token while the additional heads generate candidate tokens further ahead. These predictions form a speculative continuation that is then verified by the target model using the same verification process as other speculative decoding methods.
Because MTP reuses the target model’s existing computation and does not require loading an additional language model, its memory overhead is significantly lower than traditional draft-model approaches.
DFLASH
In Atomic Chat, DFLASH achieves up to 6× faster generation on Qwen 3.6, Gemma 4, and Kimi K2.5 in workloads where the generated blocks maintain high acceptance rates. Its performance is particularly dependent on the ability of the block-diffusion model to produce candidates that remain consistent with the target model during verification.
DFLASH predicts a block of future tokens simultaneously while the target model then verifies the entire block in a single verification step. This design reduces the sequential bottleneck of the drafting stage and allows the draft model to produce candidate sequences more efficiently. The main limitation is that the block structure is determined by the trained draft model, which limits how dynamically the draft length can be adjusted compared with token-by-token drafting methods.
Summary
Speculative decoding accelerates autoregressive generation by proposing candidate tokens with a cheap drafting mechanism and verifying them in a single target-model forward pass. In Atomic Chat, speculative decoding results in a throughput increase on the order of 30–70% from MTP and up to 6× from DFLASH — without any loss in quality.
FAQ
Does speculative decoding change the model’s output?
No. Verification uses the target model’s own conditional probabilities under a rejection-sampling rule, so the generated text is mathematically identical to standard decoding. Only the number of forward passes changes, not the tokens produced.
Does speculative decoding reduce quality or accuracy?
No. It is lossless by construction. A drafted token is kept only when it matches what the target model would have sampled anyway; the first mismatch is corrected directly from the target’s distribution. The speedup comes from doing the same work in fewer passes.
What is the acceptance rate?
The acceptance rate is the fraction of drafted tokens that survive verification — accepted tokens divided by generated tokens. It is the main determinant of how much speed you gain: higher acceptance means each verification pass advances the sequence by more tokens.
When does speculative decoding help the most?
On workloads with predictable continuations — iterating over code, reasoning models that repeat earlier thinking, and summarization that echoes its source. These produce long accepted runs. It helps least on short, highly novel, free-form text, where low acceptance can make the drafting overhead outweigh the benefit.
Why is a draft that guesses ahead faster than generating one token at a time?
Autoregressive generation is limited by memory bandwidth, not raw compute: each token requires reading the full model’s weights from memory. Verifying several drafted tokens in one batched pass reuses a single weight read for many tokens, so correct guesses are accepted at almost no additional cost.
What is the difference between a draft model and Multi-Token Prediction (MTP)?
A draft model is a separate, smaller network that must be loaded and run alongside the target. MTP instead adds prediction heads to the target model itself, so it drafts several tokens from a single forward pass with far lower memory overhead — but the model must be trained with those heads to support it.
Do I need to configure speculative decoding in Atomic Chat?
No. Atomic Chat applies speculative decoding automatically on supported models through its own tuned inference engine, using Multi-Token Prediction and DFLASH. There is nothing to enable or tune by hand.
