Continuous Batching Without Tail Spikes

Choose iteration-level admission from a production arrival replay, then protect first-token latency, fairness, cache pressure, and overload recovery.

Continuous batching raises LLM throughput only when iteration-level queue changes do not turn first-token latency into collateral damage. This guide uses an arrival-trace replay to choose the policy, token budget, and admission boundary that improve useful work without hiding p95 spikes.

The proposed method is for an engineer who can already make a model generate but needs a serving decision that survives bursty production traffic. Its deliverable is a scheduler receipt: queue state, batch composition, time to first token, inter-token delay, completion rate, and the exact rule that ships.

The operating vocabulary connects iteration-level scheduling, LLM serving throughput, dynamic batching, and tail latency; each phrase names one part of the queue contract rather than a second search intent. That vocabulary supports the article's continuous batching decision without creating a second intent.

Arrivals
Iteration gate
Token step
Latency receipt

Figure 1: Requests enter at iteration boundaries, share bounded token work, and leave with queue and latency evidence attached.

Continuous batching starts with an arrival trace

A synthetic steady stream makes every scheduler look calmer than the traffic it will actually face. The Orca OSDI paper describes iteration-granularity control and selective batching as serving mechanisms for transformer generation workloads. A queue policy should be judged on burst shape, prompt length, and cancellations together; smoothing those arrivals removes the reason to study the scheduler. A proposed review of “Continuous batching starts with an arrival trace” has four inspectable moves.

Capture timestamps before the proxy coalesces requests
Preserve prompt and output-length distributions
Mark tenant, model, and adapter cohorts
Replay at original and stressed arrival rates

The proposed evidence for continuous batching starts with an arrival trace is comparing identical trace IDs and request payloads across policies. The continuous batching starts with an arrival trace receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this continuous batching decision is benchmarking on a rectangular batch that never occurs in production. If benchmarking on a rectangular batch that never occurs in production occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is no policy comparison is valid unless the arrival trace and completion set are identical. Promotion under “no policy comparison is valid unless the arrival trace and completion set are identical” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Schedule at token boundaries

Generation requests shrink and finish at different times, so the useful scheduling boundary is the decode iteration rather than the whole sequence. The S-LoRA paper shows why heterogeneous adapter requests add memory and scheduling pressure to high-throughput serving. Iteration boundaries matter because completed rows and newly admitted rows change the token workload before every decode step. A proposed review of “Schedule at token boundaries” has four inspectable moves.

Remove completed rows before the next step
Admit waiting rows only at declared boundaries
Cap tokens rather than request count
Record the composition of every step

The proposed evidence for schedule at token boundaries is a step-by-step ledger of active sequences and issued tokens. The schedule at token boundaries receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this continuous batching decision is allowing one long sequence to occupy a slot after shorter work can join. If allowing one long sequence to occupy a slot after shorter work can join occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is the next admission may proceed only when memory and token budgets remain satisfied. Promotion under “the next admission may proceed only when memory and token budgets remain satisfied” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Plot the latency-throughput frontier

One scalar throughput number hides whether extra work came from delaying new arrivals or starving nearly finished sequences. The vLLM documentation documents a production engine where scheduler, cache, and sampling configuration are explicit operating surfaces. The most informative frontier point is the knee where a modest throughput gain starts buying a disproportionate first-token delay. A proposed review of “Plot the latency-throughput frontier” has four inspectable moves.

Report prompt and decode throughput separately
Plot median and p95 first-token time
Track inter-token delay for active streams
Include cancellations and timeouts in the denominator

The proposed evidence for plot the latency-throughput frontier is a frontier whose points share hardware, model, trace, and output set. The plot the latency-throughput frontier receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this continuous batching decision is selecting the fastest aggregate point despite a broken interaction tail. If selecting the fastest aggregate point despite a broken interaction tail occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is ship the highest-throughput point that clears every declared latency objective. Promotion under “ship the highest-throughput point that clears every declared latency objective” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Signal	Decision	Evidence
Static batch	Baseline	41 tok/s · 310 ms p95
Open queue	Reject	79 tok/s · 1.8 s p95
Bounded join	Ship	72 tok/s · 420 ms p95

Figure 2: Hypothetical worked example. These figures are illustrative, not measured production or client results. The winning policy sits on a workload-specific frontier; maximum tokens per second is not the only release objective.

Encode a bounded join policy

A small executable policy is easier to review than a serving flag whose queue semantics live in tribal memory. Executable admission logic makes skipped requests explainable and gives overload behavior a reviewable source of truth. A proposed review of “Encode a bounded join policy” has four inspectable moves.

State the maximum live-token budget
Reserve capacity for admitted sequences
Reject or defer oversized prompts explicitly
Emit a reason for every skipped arrival

The proposed evidence for encode a bounded join policy is running the policy against a fixed queue fixture. The encode a bounded join policy receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this continuous batching decision is admitting work optimistically and discovering the memory peak inside a kernel. If admitting work optimistically and discovering the memory peak inside a kernel occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is every admitted row must fit the reserved token and memory envelope. Promotion under “every admitted row must fit the reserved token and memory envelope” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Runnable artifact. Save this bounded specimen as continuous-batching-llm-inference.test.mjs and run node --test continuous-batching-llm-inference.test.mjs. Expected result: PASS: token budget bounds admission.

import assert from "node:assert/strict";
import test from "node:test";
const admit = (active, waiting, budget) => active + waiting <= budget;
test("bounds iteration admission", () => {
  assert.equal(admit(72, 24, 96), true);
  assert.equal(admit(88, 24, 96), false);
  console.log("PASS: token budget bounds admission");
});

Protect fairness across cohorts

A global queue can improve the mean while rare long prompts or cold adapters wait indefinitely. Fairness belongs in the scheduler state because a healthy global mean can coexist with one cohort aging indefinitely. A proposed review of “Protect fairness across cohorts” has four inspectable moves.

Partition results by length and tenant
Count consecutive skips per request
Set a maximum queue age
Test a burst from the smallest cohort

The proposed evidence for protect fairness across cohorts is the worst supported cohort's p95 wait and completion rate. The protect fairness across cohorts receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this continuous batching decision is letting abundant short work continuously jump ahead of expensive but valid work. If letting abundant short work continuously jump ahead of expensive but valid work occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is promote or reserve capacity before any supported cohort crosses its age limit. Promotion under “promote or reserve capacity before any supported cohort crosses its age limit” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Treat cache pressure as scheduler state

The batching decision is incomplete when it ignores block availability, fragmentation, and eviction cost. Cache preemption converts an apparent admission win into repeated prompt work, so block pressure must sit beside queue depth. A proposed review of “Treat cache pressure as scheduler state” has four inspectable moves.

Log free and reclaimable cache blocks
Attribute preemption to the triggering admission
Separate prefix reuse from fresh allocation
Replay near the memory knee

The proposed evidence for treat cache pressure as scheduler state is useful tokens completed per second at each cache occupancy band. The treat cache pressure as scheduler state receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this continuous batching decision is celebrating GPU utilization while preemption repeats prompt work. If celebrating GPU utilization while preemption repeats prompt work occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is back off admission before recomputation erases the frontier gain. Promotion under “back off admission before recomputation erases the frontier gain” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

CaptureCapture
Record arrivals, prompt sizes, and requested output limits.
ReplayReplay
Feed the same trace through every candidate policy.
MeasureMeasure
Compare TTFT, token gaps, completions, and utilization.
GateGate
Ship only the policy inside latency and fairness limits.

Figure 3: A trace replay turns scheduler tuning into a repeatable traffic experiment.

Test cancellation and overload

Real clients disconnect, retry, and exceed estimates, so the scheduler needs a truthful degraded mode. Cancellation tests reveal whether abandoned rows release both scheduler state and cache blocks before the next burst arrives. A proposed review of “Test cancellation and overload” has four inspectable moves.

Cancel queued and active requests
Inject output-length underestimates
Drive arrivals beyond sustainable service
Confirm bounded memory after cleanup

The proposed evidence for test cancellation and overload is drain time, wasted tokens, and queue recovery after the injected burst. The test cancellation and overload receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this continuous batching decision is measuring only requests that survive to a clean completion. If measuring only requests that survive to a clean completion occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is overload must shed new work without corrupting admitted streams. Promotion under “overload must shed new work without corrupting admitted streams” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Publish the scheduling receipt

The production policy should be reconstructable from a graph, a config revision, and one trace identifier. A publishable scheduler claim needs the trace identity and rejected policy points, not merely the selected configuration. A proposed review of “Publish the scheduling receipt” has four inspectable moves.

Version queue and cache settings together
Attach hardware and model identity
Name latency objectives beside charts
Keep the rejected frontier points

The proposed evidence for publish the scheduling receipt is a clean replay from the checked-in trace and configuration. The publish the scheduling receipt receipt should bind the relevant input and configuration to the observed output, then retain a deliberately rejected control so the check can prove it distinguishes a bad result.

The falsification target for this continuous batching decision is showing a heroic throughput number without the arrival process that produced it. If showing a heroic throughput number without the arrival process that produced it occurs, a favorable aggregate can still conceal the exact cohort, queue state, position, geometry, or frame that invalidates the conclusion.

The article's proposed boundary is another engineer can rerun the chosen point and explain why faster points were rejected. Promotion under “another engineer can rerun the chosen point and explain why faster points were rejected” should wait until another reviewer can evaluate the boundary from named evidence without relying on the author's authority.

Keep the boundary visible

Choose a batching policy from a production-shaped replay, not from a rectangular benchmark. Token-level admission, cache pressure, cohort fairness, and cancellation belong in the same receipt as throughput.

JP's proposed policy favors the last frontier point before interactive latency or overload recovery bends sharply, rather than the absolute throughput peak. The reason for that boundary should remain visible.

Continue through four related field notes: LLM admission control, KV cache optimization, speculative decoding, disaggregated inference. Each extends the continuous batching method without changing this article's single search intent.