Every token T_n in a standard language model depends causally on all prior tokens. That dependency is not incidental: it is load-bearing for coherence. It is also the root of the serving constraint. Understanding why the constraint exists at the architectural level, rather than as an optimization failure, is necessary context before you can reason about what approaches might address it and which cannot.
Why causal masking is not optional
Transformer-based language models are trained with a causal mask applied to self-attention. The mask prevents each position from attending to positions that appear later in the sequence. This is not an accident of implementation. It is required for the training objective to be coherent: if position i can attend to position j where j is greater than i, the model can trivially predict token i by looking at the correct answer. Causal masking enforces the constraint that each token must be predicted from its predecessors only, which is what makes the training signal real and the learned representations useful.
At inference time, this same structure means position i genuinely cannot be determined until positions 1 through i-1 are resolved. The model has not merely been trained to generate left to right: it is structurally dependent on the sequential ordering because its weights encode conditional distributions over next tokens given prior context. Position 50 of a response is a draw from a distribution conditioned on positions 1 through 49. You cannot compute that distribution without the prior 49 positions being committed to specific tokens first.
The memory bandwidth constraint on each step
Given that each forward pass must happen sequentially, what determines the minimum time per step? For large transformer models at small batch sizes, the answer is memory bandwidth, not arithmetic throughput.
Each autoregressive decoding step reads the full set of model parameters from GPU memory: all weight matrices across all attention heads and feedforward layers. That is a substantial data movement, typically on the order of hundreds of gigabytes for a production-scale model. The actual arithmetic performed on those weights, multiplying them against the current hidden state to produce the next-token distribution, is comparatively small because you are processing only one or a handful of token positions per step.
The resulting profile is that arithmetic units sit underutilized while the memory system runs near its peak transfer rate. This is what practitioners mean when they say decode is memory-bandwidth bound at low batch size. You are paying the weight-loading cost on every step regardless of how many tokens you are processing per step, and at batch size 1 that cost is entirely amortized over a single token. Larger batches amortize it over more tokens per pass, which is why throughput and latency have fundamentally different optimization targets in serving systems.
Why KV caching does not resolve the sequential constraint
KV caching is a genuine and important optimization for autoregressive serving. Once a token is generated, its key and value projections are cached and reused in all subsequent attention computations. This eliminates the need to recompute those projections in later steps. It also eliminates the need to re-attend over the full prior context from scratch on every step. The attention computation at step n costs roughly proportional to n rather than to n-squared, which is the naive cost without caching.
What KV caching does not change is the total number of forward passes required. Generating N output tokens still requires N steps. Each step still requires loading model weights through the memory system. The sequential dependency that forces N steps is not an artifact of inefficient attention computation. It is a structural property of the decoding algorithm. KV caching makes each step cheaper, but it does not reduce the number of steps. The floor remains proportional to N.
This distinction matters for how you frame latency optimization work. Teams who have invested heavily in KV cache tuning and are asking why latency is still high for long completions are running into the fundamental constraint, not a remediable inefficiency. You can approach the floor efficiently. You cannot go below it while generating autoregressively.
What the floor looks like in practice
Think about a response of around 300 tokens, which is typical for a substantive chat answer or a moderate function implementation. At good decode efficiency, each step takes some amount of time that reflects memory bandwidth and a small compute overhead. The total generation time is that per-step time multiplied by 300. The product is the floor: achievable in principle, but not improvable except by making each individual step faster or by changing what hardware you are running on.
For teams building products where users are waiting for responses, the consequence is that there is a quality ceiling on the user experience at any given output length. Longer responses cost more user patience. This shapes product decisions in ways that are not always obvious: teams end up trimming responses not because shorter is better but because waiting is worse. The latency floor of autoregressive generation is quietly influencing what products can responsibly promise users.
Attempts to work around the floor
Several approaches attempt to reduce effective latency while remaining within the autoregressive paradigm. Speculative decoding uses a small draft model to generate candidate token sequences, which the large model then verifies in a batch forward pass. When the draft model's tokens are accepted, you get multiple tokens per large-model forward pass. The gains are real but bounded: they depend on the draft acceptance rate, and the acceptance rate degrades when the distribution being sampled from shifts away from what the draft model learned. More on this in a separate post.
Quantization reduces the per-step cost by shrinking the per-parameter memory footprint, which allows more weight data to be loaded per unit time given fixed memory bandwidth. This is valuable but again moves you closer to the floor without eliminating it. You are reducing the proportionality constant, not changing the proportionality.
Continuous batching and dynamic sequence packing improve GPU utilization by mixing requests at different stages of generation in the same forward pass, amortizing the fixed per-step costs across more active requests simultaneously. These improve throughput substantially and reduce the effective cost per token, but the per-request latency for individual long completions is not improved by batching more requests together.
A different mathematical path
The interesting question is whether there is a decoding algorithm that does not inherit the sequential dependency as a hard constraint. Diffusion-style generation over discrete token spaces is one such path. Rather than resolving tokens sequentially, it initializes all positions simultaneously and refines them over K joint steps. If K scales less rapidly than N with output length, the floor is structurally lower.
This is not a claim we make carelessly. The refinement schedule K is real and adds its own cost profile. And for short outputs the advantage narrows substantially. But for the workloads where sequential latency compounds most visibly, such as long code generation tasks or document-length responses, the mathematical structure of the problem is different enough that the floor is in a different place. That is the core of what we are building at Inception: an inference runtime around a model architecture trained for iterative parallel refinement, designed to serve teams where output length is the dominant variable in serving cost.