p50 vs p99, time-to-first-token vs time-to-last-token, warm vs cold path: the choices you make when setting up an inference benchmark determine what story it tells. The problem is that many of those choices look neutral but are not.
This is not a tutorial on benchmarking tools. It is a guide to the methodological decisions that determine whether your benchmark result means something or just confirms what you already believed.
Start with what users actually experience
The first question is not "what metric should I measure?" It is "what does my user experience during generation?" The answer determines which metrics matter.
Two users with the same average latency can have meaningfully different experiences. One waits 400ms before seeing the first token, then watches the output stream at a steady rate. Another waits 50ms for the first token, then experiences an uneven stream with occasional pauses. The average end-to-end time might be identical. The experience is not.
Before you write a single benchmark harness, write down the user experience your product is promising. Then match your metrics to that promise.
The percentile problem
Why p50 lies
Median latency is a description of the typical request in a load test. It is not a description of what half your users experience, because request distributions are not symmetric. In inference serving, tail latency tends to be driven by: batch contention at peak load, cold-start penalties on the first request to a new replica, KV cache misses on unusually long prompts, and occasional garbage collection pauses in the serving framework.
These events are rare enough to stay below p95, but frequent enough that a user who sends ten requests in a session has a significant chance of hitting one. If you only report p50, you are hiding the experience of a meaningful fraction of active users.
Which percentile to care about
The right percentile depends on your product's error budget. A batch processing pipeline that runs overnight can tolerate p99 being elevated if it does not affect the next-day delivery SLA. An interactive assistant where users are waiting for responses needs p95 or p99 to stay within perceptible limits.
A reasonable baseline for interactive applications: report p50, p90, p95, and p99. The gap between p95 and p99 tells you whether your tail is driven by rare catastrophic events or by a fat distribution. Both are fixable, but differently.
Warm path vs cold path
Many inference benchmarks run against a pre-warmed serving instance: the model is loaded, the KV cache is pre-populated, and the first few requests have already run through the system. This eliminates initialization latency, JIT compilation delays, and first-request outliers.
Whether this is right depends on your product's request pattern. A product with continuous high-traffic has mostly warm-path behavior; a developer API with bursty usage sees cold-path behavior regularly.
The failure mode is benchmarking under warm conditions to determine whether latency is acceptable, then deploying to a product with cold-start patterns and discovering that users who arrive during a low-traffic period see dramatically higher latency than the benchmark suggested.
The fix is to measure both explicitly. Run a warm benchmark for sustained-load characterization. Run a cold benchmark for first-request-after-idle characterization. Report them separately. Do not average them.
Concurrent load and queue effects
Single-request benchmarks tell you about the latency of one isolated request. They do not tell you anything about how your system behaves under realistic concurrency.
Inference serving systems have batch schedulers that trade per-request latency for throughput. At low concurrency, each request may execute immediately. As concurrency increases, requests queue up, batch together, and share GPU resources. The latency of any individual request depends on what else is running at the same time.
A benchmark that measures single-request latency then projects to production load is methodologically wrong. The projection does not hold because the queueing dynamics at production concurrency are absent in the single-request test.
Benchmark at realistic concurrency levels. For a product where 10 users might be generating simultaneously, measure at 1, 5, 10, and 20 concurrent requests. The shape of the latency curve as concurrency increases tells you where your system saturates and by how much.
Input and output length distributions
Latency in language model inference depends on both input length (prefill cost) and output length (decoding cost). These are not the same. Prefill scales with prompt token count and is often parallelizable across the prompt. Decoding scales with output token count in ways that depend on your decoding architecture.
A benchmark that uses a fixed prompt of 100 tokens and measures latency tells you about that specific input shape, not about your product's actual prompt distribution. If your users send prompts that range from 50 to 2000 tokens depending on how they use the product, your benchmark needs to sample from that distribution.
Similarly: benchmarking against requests that produce 50-token outputs then projecting to a product where outputs average 300 tokens will underestimate serving cost. Profile your actual request distribution first, then build synthetic load that matches it.
What a parallel decoding benchmark requires specifically
If you are evaluating parallel decoding alongside autoregressive generation, the comparison requires extra care. The two approaches have different latency profiles across the output length distribution.
Autoregressive generation scales linearly with output length. Parallel decoding scales with the number of refinement steps K, which grows more slowly than N for longer outputs. A benchmark that uses only short outputs will show one comparison; a benchmark that uses only long outputs will show a different one. Neither is representative of a mixed workload.
The correct methodology: measure both approaches across your actual output length distribution, weight results by the frequency of each length range, and compute a weighted composite. A benchmark that shows results only at a fixed output length is not a fair evaluation of either approach for real workloads.
This is not a caveat we add reluctantly. It is the methodological requirement for comparing decoding approaches honestly. Any benchmark that does not account for output length distribution is measuring something, but not the thing you need to know to make an architecture decision.
The quality dimension
Latency benchmarks without quality checks can mislead in a specific direction: you may measure that approach A produces outputs faster than approach B, without checking whether the outputs are equivalent in quality at those latency points.
This matters for parallel decoding with fewer refinement steps. Running K steps instead of K+N steps is faster. Whether the quality difference is acceptable depends on your use case, not on a universal threshold. A benchmark that only measures latency and ignores quality is not a complete evaluation.
The right approach: run quality evaluation (via human review, automated scoring, or task-specific metrics appropriate to your domain) at the same time as latency measurement. Understand the quality-latency tradeoff at different step counts, then choose the operating point that meets your quality floor.
Reporting benchmarks to others
When you share benchmark results with your team, your investors, or your customers, the summary number conceals the methodology. The methodology is the part that determines whether the number is meaningful.
A good benchmark report states: what percentiles were measured, whether the path was warm or cold, what concurrency level was tested, what the input and output length distribution was, and what quality evaluation was run alongside the latency measurement.
A headline number without that context is not dishonest, but it is incomplete. The person reading it will fill in assumptions from their own experience, and those assumptions may not match your test conditions.