LLM Serving Goodput: Count Only SLO Wins

A server can report record token throughput while most requests miss the latency users were promised. LLM serving goodput counts only requests that clear declared latency gates and keeps misses, failures, and incomplete work visible in the same capacity receipt.

LLM serving goodput exposes hidden failure

Imagine a 60-second test that completes 120 requests and emits 24,000 tokens. The dashboard celebrates 400 output tokens per second. Yet 54 requests delivered a first token after the product's one-second promise, 18 stalled between tokens, six failed, and five were still running when the window closed. Raw throughput describes work performed; it does not describe useful service delivered on time.

LLM serving goodput asks a narrower question: how many requests per second completed while satisfying every declared service-level objective? If 42 requests met the time-to-first-token, time-per-output-token, and end-to-end gates in that window, goodput is 0.7 qualifying requests per second. The other outcomes do not vanish. They remain named misses, failures, or censored work in the same receipt.

This distinction matters near overload. A scheduler may keep devices busy and increase raw token throughput while queues make the product slower. Continuous batching for LLM inference explains the scheduling frontier; a goodput SLO turns that frontier into a product-facing acceptance test.

Goodput is not a synonym for speed and does not measure answer quality, safety, or cost. It is a measurement contract over one workload, one window, and explicit latency gates. Keeping TTFT and TPOT separate prevents a fast first token from concealing a stalled stream. The contract is valuable precisely because it refuses to let completed-but-late work decorate the success count.

Freeze the goodput SLO before the run

Write the measurement points before collecting a trace. Let arrival be the instant the load generator makes a request eligible for service, first token be the first model token observed at the declared client or server boundary, and completion be the final model token or terminal response. Time to first token is first token minus arrival. End-to-end latency is completion minus arrival. For an output with more than one token, time per output token is completion minus first token divided by output tokens minus one.

The equality rule belongs in the LLM serving goodput contract. This tutorial treats a value equal to its limit as passing: TTFT less than or equal to its gate, TPOT less than or equal to its gate, and end-to-end latency less than or equal to its gate. Changing less-than-or-equal to strict less-than can flip boundary rows, so the receipt records it.

Do not force every request into one SLO class. An interactive 80-token answer, a batch extraction, and a long-form completion may have different promises. Assign the class before the result arrives, store its gates with the trace, and report each class separately before any roll-up. Otherwise a generous long-output threshold can conceal misses in an interactive cohort.

Metric names are not fully standardized across serving tools. Publish formulas and observation boundaries rather than assuming that two fields named TTFT or inter-token latency are identical. Server-Timing for AI latency is useful when the browser-facing boundary must stay aligned with server phases.

Compute TTFT and TPOT from timestamps

A defensible LLM serving benchmark keeps the raw evidence needed to recompute every derived field. That evidence is the foundation of LLM serving goodput. At minimum, each row needs a stable request identifier, arrival time, status, output-token count, first-token time when one exists, completion time when one exists, SLO class, and relevant workload labels such as prompt length and cache state. Store milliseconds or another explicit unit consistently; never infer the unit from magnitude.

For a successful multi-token stream, TTFT equals firstTokenMs minus arrivalMs. TPOT equals completedMs minus firstTokenMs divided by outputTokens minus one. End-to-end equals completedMs minus arrivalMs. A one-token output has no interval between output tokens, so TPOT is unavailable, not zero. It can pass TTFT and end-to-end while the TPOT gate is marked not applicable under a declared rule.

Count model tokens, not arbitrary network chunks. One server-sent event can contain several tokens, and buffering can place several chunks at the same timestamp. If only chunk timestamps exist, label the result chunk cadence; do not rename it TPOT. Tokenization and stream framing are part of the measurement receipt.

Reject non-finite timestamps, completion before arrival, first token before arrival, completion before first token, negative token counts, duplicate identifiers, and a successful completion with no tokens. Individually finite endpoints are insufficient: subtraction can still overflow. The lab bounds timestamp magnitude to 10^15 milliseconds, bounds window and request spans to 86,400,000 milliseconds, and rejects any non-finite derived duration, rate, or share before returning a receipt. Keeping invalid rows outside both the pass count and the valid denominator is acceptable only when their count and reasons remain visible. Silent cleanup makes goodput look better by erasing difficult evidence.

Keep failures and window edges visible

A request that times out, is cancelled after admission, or ends with a server error consumed capacity without delivering a qualifying completion. Keep it in the workload count and report it as failed with its terminal reason. Do not give it synthetic completion or first-token timestamps merely to make formulas run. Failure is a verdict category, not a malformed success.

The declared measurement window creates two other edges. A request may arrive during the window and still be running at its end. That row is censored: the final latency is unknown. A request may have arrived before the window and complete inside it. Mixing arrival-window and completion-window accounting can create impossible rate comparisons. This tutorial uses arrivals within the window as the cohort, then reports completed passes, completed misses, failures, and in-flight censored rows.

Warm-up should be explicit. If the experiment excludes compilation, model loading, or cache priming, record the warm-up interval and exclude all arrivals from it by rule, not by inspecting slow results. If warm-up reflects the user path, include it. Cache state deserves the same treatment: cold, prefix-hit, and mixed workloads are distinct fixtures rather than a hidden optimization.

The denominator depends on the question. LLM serving goodput rate is passing completions divided by window seconds. Pass share can be passes divided by all valid cohort arrivals, with censored and failures shown. Publishing both prevents a short window with many unfinished requests from appearing healthy. The figure below turns those categories into visible lanes rather than a footnote.

Replay a production-shaped load

Closed-loop tests, where each virtual user waits before sending again, can reduce offered load when latency rises. That feedback may conceal the overload knee. For capacity work, an open arrival process or a recorded arrival schedule is usually more revealing: requests continue to arrive according to the declared pattern while the server queues, rejects, or recovers. The arrival distribution, burst size, and random seed belong in the receipt.

Represent prompt and output lengths with distributions, not one convenient average. Include short interactive requests, longer context, and the tail that the product actually admits. If speculative decoding, prefix caching, or disaggregated prefill is enabled, record its configuration and cohort. Disaggregated LLM inference can shift TTFT and TPOT differently, which is exactly why separate gates matter.

Replay at several offered rates long enough to observe a stable region, an approach to saturation, and a recovery interval. At each point, publish offered requests per second, admitted requests, raw completed throughput, LLM serving goodput, failures, queue delay when available, token distributions, and quality results from a separate evaluation. A synthetic sweep is an experiment plan, not a production forecast.

Keep the client honest too. Its clock, connection pool, timeout, and ability to consume streams can create apparent server latency. Co-locate or synchronize measurement where appropriate, and state whether arrival means scheduled, socket-written, server-received, or admitted. A capacity number without that boundary cannot be compared safely.

Build a reproducible goodput receipt

The downloadable LLM serving goodput lab accepts a JSON object containing a declared window, explicit SLOs, and at most 500 request rows in 64 KiB. It validates finite monotonic timestamps and status-specific fields before computing anything. Every valid row receives raw TTFT, TPOT, and end-to-end values where defined, plus a verdict and ordered miss reasons. Failed and censored rows remain separate from latency misses.

The aggregate receipt includes schema version, equality rule, window duration, gates, workload labels, cohort counts, passing completions, goodput per second, pass share, and reason counts. Export the receipt beside the original trace or its privacy-safe hash. That is enough for another engineer to inspect why one number changed without requiring access to a serving cluster.

Run a sensitivity sweep after the primary result, never instead of it. Recalculate the same fixed trace under neighboring TTFT, TPOT, and end-to-end gates to see whether the conclusion depends on a single threshold. Label each alternative hypothetical. Choosing the threshold that maximizes a completed run is gaming the contract.

The fixtures include equality boundaries, a one-token completion, a timeout, malformed ordering, warm-up, cold and warm cache labels, and an in-flight request. A mutation check removes known misses and proves that the apparent pass share rises. This is an intentional anti-cheating test: if dropped rows do not change the headline, the denominator or reporting path is wrong.

Read the latency-constrained throughput frontier

Plot offered load on the horizontal axis and several outcomes on the vertical axis. Raw completions often rise toward device saturation. LLM serving goodput may rise, flatten, and then fall as queueing pushes requests beyond their latency gates. The divergence is the overload knee: more work is being processed, but less promised service is being delivered.

Do not collapse the frontier to one winning configuration. Publish latency-constrained throughput beside p50 and tail latencies, failures, answer quality, and cost. A scheduler that improves goodput by rejecting most hard requests may be useful admission control, but only if the rejected cohort and product behavior are acceptable. LLM admission control covers that decision surface.

The synthetic values in the figure are illustrative and labeled as such. Results in systems research, including DistServe at OSDI ’24, depend on the evaluated models, hardware, workload, and SLO definitions. The vLLM benchmark CLI provides concrete benchmark controls, but an option name does not replace a product measurement contract. The paper on revisiting SLO and system metrics is a useful warning against treating one aggregate as the system.

Goodput says nothing about whether the answer was correct or safe. Pair capacity evidence with a fixed quality bank, and refuse configurations that purchase latency by damaging the output.

Turn the receipt into an admission decision

Choose the operating point before launch with an explicit release gate: for the declared workload and SLO classes, LLM serving goodput must exceed the required arrival rate with a stated safety margin, failure share must stay below its ceiling, tail latency must recover after the planned burst, and quality must pass independently. The exact thresholds are product decisions, not universal constants supplied by this article.

When the gate fails, the receipt suggests different actions. Predominant TTFT misses point toward queueing, prefill, cold starts, or admission. TPOT misses point toward decode contention or streaming behavior. End-to-end-only misses can expose long outputs or a promise incompatible with the workload. Failures and censored growth demand their own investigation. AI agent load testing with Little’s Law helps connect concurrency, arrival rate, and residence time without pretending every workload is stationary.

Use the same receipt in monitoring. Version the SLO class, sample traces without dropping failures, and alert on sustained goodput deficit rather than only GPU utilization. A rollback should restore the last configuration that passed the complete contract, not merely the one with lower average latency.

LLM serving goodput earns trust when every exclusion is visible: measurement boundary, workload, warm-up, cache state, failures, in-flight work, formulas, and gate versions. It is not a flattering throughput label. It is a compact proof that admitted work became usable service on time.

Runnable local artifact — The lab analyzes supplied timestamps; it does not benchmark a model or recommend a universal latency SLO.

Validate each request, preserve failed and censored outcomes, compute TTFT/TPOT/E2E at declared measurement points, and divide passing completions by the declared window.