Chunked Prefill Without Decode Jank

Chunked prefill prevents one long prompt from monopolizing an LLM serving step, but a careless chunk budget can still starve decode work or stretch time to first token. This guide builds a mixed-traffic replay that makes the trade visible before the scheduler reaches production.

The reader should leave with one answer: how to choose a prefill chunk size from prompt arrivals, decode cadence, GPU occupancy, and tail latency rather than from a default flag.

The supporting vocabulary is decode-first scheduling, streaming latency split, inter-token latency, LLM scheduling. Each term serves the same search intent: choose a prefill chunk budget that protects decode responsiveness under mixed prompt lengths.

My position is that chunking is a latency-control instrument, not a throughput toggle. The smallest useful policy is one that names who can wait, for how long, and under which workload shape.

1. Prompt arrivals
2. Prefill slices
3. Decode lane
4. Latency gate

Chunked prefill starts with mixed traffic

A benchmark made only of identical prompts removes the interference the policy exists to control. The vLLM chunked-prefill documentation documents chunked prefill as a serving feature and exposes its scheduling controls. In this chunked prefill method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Sample short, medium, and long prompt cohorts
• Preserve output-length correlation
• Replay burst and steady periods
• Keep arrival order visible

The useful measurement is per-cohort queue time plus global GPU utilization. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is a long prompt pauses visible decode progress. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is ship only against a production-shaped arrival trace. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives chunked prefill a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Separate first-token and decode pain

TTFT and inter-token delay belong to different user moments and should not collapse into one latency average. The Sarathi-Serve paper describes chunked prefills as part of a stall-free LLM inference design. In this chunked prefill method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Record enqueue-to-first-token time
• Record every decode gap
• Split p50, p95, and maximum
• Retain request-level traces

The useful measurement is a latency frontier by cohort. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is a favorable mean hides frozen streaming responses. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is neither protected metric may cross its explicit budget. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives chunked prefill a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Sweep one prefill chunk size

Change only the token budget so the result explains the scheduler rather than a pile of coupled tuning. The Orca paper establishes iteration-level scheduling as a foundation for modern LLM serving. In this chunked prefill method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Choose a logarithmic token ladder
• Warm the model before capture
• Use the same arrival trace
• Repeat enough runs for stable tails

The useful measurement is throughput, TTFT, ITL, and occupancy at each budget. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is the largest slice wins throughput by starving decode. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is prefer the knee of the latency-throughput curve. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives chunked prefill a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Option	Observed signal	Verdict
Unchunked	decode gaps widen	baseline
Large slices	occupancy rises; tails wobble	reject
Bounded slices	decode cadence holds	ship

Model the scheduling queue

A trace should expose which requests joined each step and how much token work each consumed. In this chunked prefill method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Log active decode sequences
• Log admitted prefill tokens
• Mark evictions and cancellations
• Snapshot free KV capacity

The useful measurement is step-level token composition. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is queue state cannot explain a tail spike. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is reject any run whose scheduler state is not reconstructable. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives chunked prefill a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Test the budget invariant

A bounded unit test catches accidental over-admission before load tests make the mistake noisy. In this chunked prefill method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Encode remaining step capacity
• Admit decode work first
• Slice the next prefill
• Assert the sum never exceeds budget

The useful measurement is maximum scheduled tokens per iteration. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is prefill admission exceeds the declared cap. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is the invariant must pass for empty, full, and mixed queues. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives chunked prefill a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Runnable artifact. Save this bounded check as chunked-prefill-scheduling.test.mjs and run node --test chunked-prefill-scheduling.test.mjs. Expected output: PASS: prefill stays inside step budget.

import assert from "node:assert/strict";
import test from "node:test";
const slice=(budget,decode,prefill)=>Math.max(0,Math.min(prefill,budget-decode));
test("bounds prefill",()=>{assert.equal(slice(256,192,200),64);assert.equal(slice(256,256,20),0);console.log("PASS: prefill stays inside step budget");});

Protect fairness without hiding policy

Short prompts may jump ahead, but the exception needs an age limit so long prompts still progress. In this chunked prefill method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Track queue age
• Set a maximum bypass count
• Measure tokens advanced per cohort
• Flag starvation explicitly

The useful measurement is wait-time distribution and progress rate. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is one cohort makes zero progress during a burst. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is no eligible request may exceed the published age boundary. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives chunked prefill a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

1. ReplayReplay

   Use real prompt and output-length distributions.

2. SliceSlice

   Sweep token budgets without changing other controls.

3. ObserveObserve

   Read TTFT, inter-token gaps, occupancy, and fairness.

4. GateGate

   Choose the smallest stable latency frontier.

Read GPU utilization as context

High utilization is useful only when it does not buy work by violating the streaming experience. In this chunked prefill method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Correlate kernel gaps with queue state
• Measure occupancy consistently
• Keep clock and power mode fixed
• Compare useful completed tokens

The useful measurement is completed tokens inside latency SLOs. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is busy hardware produces late answers. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is optimize qualified throughput rather than raw activity. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives chunked prefill a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Publish the scheduler recipe

The release artifact should let another engineer repeat both the good run and the rejected control. In this chunked prefill method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Save the arrival trace hash
• Save engine and model versions
• Save chunk and queue settings
• Save cohort scorecards

The useful measurement is clean-run reproduction of the chosen frontier. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is a version change silently moves the frontier. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is rebenchmark when model shape, engine, or traffic mix changes. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives chunked prefill a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

The useful version is bounded

Choose the chunk budget from the mixed workload, not from folklore. The useful result preserves streaming cadence while allowing long prompts to advance in visible, bounded slices.

Keep the rejected large-slice run beside the selected configuration. That contrast explains why the policy exists and gives future engine changes a concrete regression target.

Continue with continuous batching, disaggregated LLM inference, KV-cache optimization, deploy readiness for product UI. Those field notes deepen adjacent implementation choices without turning this page into several articles at once.