Speculative decoding is a genuine improvement to autoregressive serving. It is not a workaround or a hack. It is a mathematically sound technique that produces output samples equivalent in distribution to the target model, while reducing the number of large-model forward passes required in expectation. Teams running high-quality large language models for latency-sensitive tasks should understand it and probably use it. But it has a ceiling, and understanding where that ceiling comes from is the prerequisite for evaluating whether an approach like parallel diffusion decoding addresses something categorically different or just solves the same problem a different way.
How speculative decoding works
The core idea: a small draft model generates a fixed-length sequence of candidate tokens at each step, typically several tokens per draft. The large target model then evaluates all those candidate tokens in a single forward pass by processing the full batch of candidates simultaneously. If the target model agrees with the draft model's token at each position up to some cutoff, those tokens are accepted. Where the target model disagrees, the disagreement triggers a rejection-sampling correction that guarantees the accepted sequence comes from the target model's true distribution.
The efficiency gain comes from the fact that verifying a batch of draft tokens costs roughly the same as generating a single target-model token. So if the draft model proposes several tokens and the target model accepts most of them, you have generated more tokens per target-model forward pass than you would have in standard autoregressive generation. Fewer large-model passes means lower total wall-clock time for a given output length.
The guarantee that the output distribution is equivalent to the target model is the critical property. Speculative decoding is not an approximation. When you reject and correct, the correction preserves the exact target distribution. This is what makes it safe to use in production without worrying that you have degraded the model's output quality.
The acceptance rate determines everything
The practical efficiency of speculative decoding is dominated by the acceptance rate: the fraction of draft tokens the target model accepts per pass. A high acceptance rate means many tokens per target-model pass and substantial latency reduction. A low acceptance rate means frequent rejections, few tokens accepted per pass, and diminishing gains from the technique.
The acceptance rate depends on how well the draft model's distribution matches the target model's distribution over the specific token being predicted. When the draft model and target model agree on the likely next token, acceptance is high. When they disagree, the rejection mechanism kicks in and the efficiency benefit of that draft step disappears. Since the correction step still takes time, a very low acceptance rate can leave you worse off than standard autoregressive decoding with no draft model at all.
This is the first natural ceiling: the quality of the draft model relative to the target model over the specific distribution you are sampling from. You cannot increase the acceptance rate beyond what the draft-target model pair naturally achieves without changing one or both models. And the draft model is typically orders of magnitude smaller than the target model, which puts a fundamental limit on how closely it can match the target model's distribution across diverse inputs.
Distribution shift degrades acceptance rate
The acceptance rate is not a fixed property of a draft-target pair. It shifts with the input distribution. A draft model trained on general text achieves one acceptance rate on general text and a potentially very different acceptance rate on specialized code, on non-English text, or on prompts that push toward the tail of the target model's distribution.
This matters for production systems. A team serving a specialized domain, such as medical note generation or structured data extraction, may find that the draft model's acceptance rate on their production distribution is substantially lower than what was measured in evaluation. The result is that the effective latency reduction they see in production is less than what benchmarks suggested. Speculative decoding's benefits are inherently tied to the alignment between the draft model's training distribution and the deployed inference distribution. Drift in one means drift in the other.
There are ways to mitigate this, including domain-specific fine-tuning of draft models and adaptive draft sequence length tuning based on observed acceptance rates. These improve the situation but do not eliminate the sensitivity. The ceiling imposed by distribution shift is structural, not a bug to fix.
The sequential constraint remains
Even at high acceptance rates, speculative decoding does not change the fundamental structure of autoregressive generation. Tokens are still produced left to right. The output at position i still causally depends on all prior positions. Each accepted token in a speculative pass is accepted in order: you cannot accept token at position i+5 while holding open the token at position i+2. The sequential causal dependency runs through the acceptance process just as it runs through standard decoding.
What speculative decoding changes is the number of large-model forward passes per output token, in expectation. What it does not change is the sequential ordering constraint. The position-by-position ordering is preserved by construction, since that is what guarantees distributional equivalence with the target model. This is not a design limitation you can optimize away. It is the requirement for the technique to be mathematically valid.
The ceiling, stated precisely: speculative decoding can reduce the effective number of large-model forward passes required, but it cannot reduce below approximately 1 pass per output token in the worst case and cannot decouple total generation cost from output length in the way that parallel refinement can. You can approach the autoregressive floor more cheaply through speculative decoding, but you cannot break through it.
Where parallel decoding starts
Parallel diffusion decoding approaches the problem differently. Instead of optimizing the number of large-model passes per sequential token, it changes the direction of generation: all output positions are initialized simultaneously and refined over K joint steps. The output length N does not directly set the number of passes. The refinement schedule K does.
This is a categorical difference from speculative decoding. Speculative decoding is an optimization applied on top of the autoregressive algorithm. Parallel diffusion decoding is a different algorithm. The sequential causal dependency that speculative decoding still respects is the dependency that parallel diffusion decoding does not have in its inference loop.
We want to be clear about what this does and does not mean. Parallel diffusion decoding requires a model trained specifically for iterative masked token refinement. You cannot take a standard autoregressive model and apply a diffusion-style inference loop to it. And the K-step refinement process has its own cost structure: K forward passes, each touching all N positions, rather than N forward passes each touching one or a few positions. The question of which is cheaper depends on the specific K vs N ratio for your task, and on how well the model has learned to converge within K steps.
For short completions where N is small, speculative decoding applied to a good autoregressive model may well outperform parallel diffusion decoding at a given quality level. For longer completions where N is large, the refinement schedule K becomes increasingly favorable as the asymmetry between K and N grows. The break-even point is empirical rather than universal.
Choosing between them for your workload
If you are running a large autoregressive model and are evaluating speculative decoding: the key diagnostic is your acceptance rate on your production distribution, not on held-out benchmarks. Measure it in your actual serving environment with your actual request distribution. If acceptance rate is stable and high for your use case, speculative decoding is a sound technique and the engineering investment to deploy a draft model is justified.
If you are evaluating parallel diffusion decoding: the relevant question is whether your workload falls in the output-length range where K is substantially less than N. Code generation tasks producing multi-hundred-token function implementations are a better fit than tasks generating brief classification labels. The tradeoff also depends on whether streaming character by character is a hard product requirement, since that is harder to support natively with parallel refinement than with autoregressive generation.
The honest assessment is that these are complementary tools with different performance profiles. Speculative decoding is mature, well-understood, and compatible with existing autoregressive models. Parallel diffusion decoding is earlier in its production maturity curve but addresses a different portion of the latency problem, specifically the floor imposed by sequential output dependency, rather than optimizing the cost of approaching that floor.
At Inception we have built our runtime around the parallel refinement approach because we believe the output-length-independence of the generation cost structure is the more important long-run property for the workload types we care about. We are not saying speculative decoding is wrong. We are saying it is the right answer to a different question.