Research and engineering
Autoregressive generation is elegant but expensive. Here is the mathematical intuition behind why initializing all output positions simultaneously, then refining them jointly, produces a different cost structure.
Every token T_n in a standard language model depends causally on all prior tokens. That dependency is not incidental: it is load-bearing for coherence. It is also the root of the serving constraint.
Diffusion models for images work in continuous pixel space. Text is discrete. This post walks through the formulation that makes diffusion-style generation work over token vocabularies.
Speculative decoding is a clever trick: use a small draft model to propose tokens, verify with the large model in parallel. It helps. But it is still fundamentally sequential at the output level.
The difference between a tolerable product and a frustrating one is often 200ms and where that 200ms lives in the request pipeline.
Your inference bill is a function of tokens generated per request, requests per second, and the per-token cost of your model. Most teams optimize the wrong variable.
Parallel decoding is not universally better. It is better for specific workload shapes. This is a candid breakdown of where the tradeoff is favorable and where it is not.
p50 vs p99, time-to-first-token vs time-to-last-token, warm vs cold path. The choices you make when setting up a benchmark determine what story it tells.
Large language model inference is usually memory-bandwidth bound at small batch sizes and compute-bound at large ones. The crossover point determines your architecture options.
Users experience time-to-first-token and the streaming rate as separate things. They care about both, but differently.
You cannot independently maximize output quality, minimize latency, and reduce cost in a deployed language model serving system. Understanding how the three constrain each other is the first step.
When speculative decoding reports a speedup ratio, that number depends heavily on the acceptance rate of the draft model. Here is why that number is sensitive to distribution shift.