Large language model inference is usually memory-bandwidth bound at small batch sizes and compute-bound at large ones. The crossover point determines your architecture options. If you do not know which regime your serving environment is in, you cannot predict how changes to batch size, model precision, or decoding strategy will affect latency.
Arithmetic intensity: the right frame
The relevant concept here is arithmetic intensity: the ratio of floating-point operations to bytes transferred from memory per operation. Every GPU has two peak rates: a peak FLOPS rate for compute, and a peak memory bandwidth rate for data movement. Which one becomes the bottleneck depends on the arithmetic intensity of the workload you run on it.
If your workload has low arithmetic intensity (few operations per byte moved), memory bandwidth saturates before the compute cores are fully utilized. If it has high arithmetic intensity (many operations per byte), compute saturates first. The ridge point is the arithmetic intensity at which both peak rates would saturate simultaneously. Workloads below the ridge are bandwidth-bound; workloads above it are compute-bound.
For a given GPU, the ridge point is a fixed hardware property: peak FLOPS divided by peak memory bandwidth. For a given operation, the arithmetic intensity is a function of the computation pattern and the data layout.
Where autoregressive decoding lands on this spectrum
During the autoregressive decoding phase (as opposed to the prefill phase), each forward pass generates exactly one token. That means: load the full model weights from GPU memory, perform a matrix-vector multiply (not a matrix-matrix multiply), produce one row of output.
Matrix-vector multiply has low arithmetic intensity compared to matrix-matrix multiply. For a weight matrix of size M x N, a single matrix-vector operation performs O(M * N) operations on O(M * N) bytes. Contrast with batched matrix-matrix multiply where you amortize the same weight read across B input vectors: O(B * M * N) operations on O(M * N) weight bytes plus O(B * N) input bytes. At large B, the arithmetic intensity approaches the compute-bound regime.
At batch size 1, autoregressive decoding is almost entirely bandwidth-bound. You load the model weights for every single token, running the compute cores at a small fraction of their peak utilization while memory bandwidth is the limiting resource.
The batch size crossover
As you increase the number of concurrent requests being decoded simultaneously (the decode batch size), the computation transitions from matrix-vector to matrix-matrix form. The same weight bytes are now read once per step but multiplied against a batch of B input vectors. The arithmetic intensity climbs toward the ridge point.
At some batch size, you cross from bandwidth-bound to compute-bound. That crossover batch size is a property of the specific model, the hardware, and the precision of the weights. For modern GPU hardware and 7-13 billion parameter models in FP16, the crossover tends to happen somewhere in the range of dozens to low hundreds of concurrent decode requests, though the specific value varies significantly with hardware generation and model architecture.
Why does this matter? Because the latency implications are different on each side of the ridge. In the bandwidth-bound regime, improving latency requires reducing memory traffic: smaller models, quantized weights, or shorter sequences. In the compute-bound regime, latency is limited by FLOPS utilization, and the strategies shift to maximizing compute efficiency.
Prefill is a different story
Prefill (processing the input prompt) is almost always compute-bound even at batch size 1. The operation is a matrix-matrix multiply over the full prompt length: for a prompt of L tokens, the input matrix has shape L x d_model, and each weight is multiplied against all L rows simultaneously. Arithmetic intensity is much higher than in single-token decoding.
This is why prefill and decode have different optimization profiles. Prefill optimizations target FLOPS utilization: fused kernels, flash attention to reduce HBM reads, efficient tiling. Decode optimizations at small batch sizes target bandwidth: weight quantization to reduce bytes transferred per weight, KV cache compression to reduce the per-step memory footprint, and batching strategies to amortize weight reads.
A serving system that optimizes only for prefill speed will underperform on decode-heavy workloads. A system that only optimizes for decode will waste GPU time on compute-heavy prefill passes. Understanding which phase dominates your workload shapes which optimizations are worth engineering effort.
What parallel decoding changes about this picture
Parallel decoding changes the decode-phase arithmetic by operating on all output positions simultaneously rather than one at a time. Each refinement step processes a full batch of output positions, transforming the operation from matrix-vector toward matrix-matrix. This shifts the arithmetic intensity upward relative to single-token autoregressive decoding.
The practical effect is that parallel decoding may be less bandwidth-bound per step than autoregressive decoding at comparable concurrency levels. This does not mean it is always faster; it means the binding constraint is different. A serving environment where bandwidth was the bottleneck may find that parallel decoding changes the bottleneck to compute, which then requires different hardware and kernel optimizations.
We do not claim this is universally better. For bandwidth-saturated serving environments, shifting to a more compute-intensive workload may require rebalancing GPU allocation. The point is that the roofline analysis changes, and you should understand how before committing to an architecture.
Identifying your bottleneck in practice
The practical way to determine whether your inference run is bandwidth or compute bound is to use GPU performance counters. Most GPU profiling tools expose memory bandwidth utilization and compute utilization as separate metrics. If memory bandwidth utilization is near peak while compute utilization is low, you are bandwidth-bound. If compute utilization is near peak while memory bandwidth headroom remains, you are compute-bound.
A quicker heuristic: run your workload at batch size 1 and at a 4x higher batch size. If latency per request scales nearly linearly with batch size in the high-batch case, you are compute-bound and adding requests costs proportional latency. If latency stays roughly flat as you increase batch size, you are still bandwidth-bound and the weight reads are being amortized without saturating compute.
Neither answer is inherently a problem. The problem is not knowing which one you are in, then applying optimizations designed for the wrong regime.
A note on quantization
Weight quantization (moving from FP16 to INT8 or INT4 weights) primarily reduces the memory bandwidth requirement per weight read. In the bandwidth-bound regime, this has a meaningful impact on decode latency because weight reads are the binding constraint. In the compute-bound regime, quantization reduces memory traffic but does not directly address the compute bottleneck, so the impact on latency is smaller.
This is why quantization benchmarks can look very different at different batch sizes. At batch size 1, INT4 quantization can significantly reduce decode latency because you are bandwidth-bound and you just cut the weight transfer cost substantially. At high batch sizes where you are compute-bound, the same quantization shows more modest latency gains.
Teams that benchmark quantization only at batch size 1 and then deploy to a high-concurrency production environment are measuring the wrong operating point.