The mechanics
How diffusion-style parallel decoding differs from autoregressive generation
A precise look at why sequential decoding creates serving constraints and how running all output positions in parallel changes the arithmetic.
The sequential constraint
Autoregressive models emit one token per forward pass
Every token Tn in a standard autoregressive model depends causally on all prior tokens T1...Tn-1. The forward pass cannot be parallelized across output positions.
Generating a 200-token response requires 200 sequential forward passes through the full model. Serving budget per request therefore scales linearly with output length, regardless of hardware utilization.
At low batch sizes, this is a memory-bandwidth problem: the model weights are read from HBM for every token pass, even when the arithmetic could be done faster. The dependency chain is the fundamental constraint.
Autoregressive constraint
P(Tn | T1, ..., Tn-1)
Each position conditions on all prior positions. No two output tokens can be generated in the same forward pass.
Latency(L) = L x ttoken
Latency scales linearly with output length L. Halving L halves serving cost; nothing else does.
The diffusion alternative
Initialize all positions simultaneously, then refine jointly
Diffusion-style decoding begins with all output positions initialized simultaneously. Each forward pass refines every position in parallel, progressively improving the joint probability of the full sequence.
Generation completes after K refinement steps rather than N individual token passes. The value of K is bounded by the sequence refinement schedule, not by output length. A 200-token response and a 500-token response may both require the same K steps to converge.
This changes the cost structure: the dominant variable shifts from output length to refinement depth K. Workloads where K is much smaller than typical output lengths see a direct reduction in required forward passes per request.
Autoregressive
Diffusion (step 2 of K)
Illustrative. Step-count comparison, not to scale, no measured performance claims. Left: autoregressive chain, each position waits for all prior positions. Right: diffusion refinement, all positions processed simultaneously per step.
Where parallel decoding is a natural fit
Better fit
- Long-form completions (100+ output tokens)
- Full-response chat turns
- Document and code generation pipelines
- Workloads tolerating K-step latency profile
Less suited
- Ultra-short completions (token count less than K steps)
- Streaming character-by-character display requirements
- Tasks where each output word conditions on a user interaction