Disaggregated Prefill for Predictable LLM Latency

Disaggregated prefill can protect LLM decode latency from prompt-processing bursts, but it also adds a KV cache transfer and another queue. This guide compares aggregated, chunked, and separated serving on one declared workload, exposes every modeled assumption, and exports a topology receipt to validate on real infrastructure.

Define the disaggregated prefill decision

Disaggregated prefill separates prompt processing from token generation so each phase can be scheduled on workers tuned for its own compute and memory shape. The useful question is not whether separation sounds more scalable. It is whether a measured request mix can absorb queueing and KV cache transfer while improving the latency objective that matters.

Start with four quantities: prompt tokens, expected output tokens, time to first token, and time per output token. Add arrival rate, concurrency, service-level targets, and the network path between worker pools. Those TTFT and TPOT measurements are the product-facing coordinates for LLM serving latency. A short prompt with a long answer stresses decode. A long document with a short answer stresses prefill. One fleet label hides that difference.

This guide builds a planning ledger rather than a GPU benchmark. Its values are declared inputs or deterministic arithmetic. Hardware, kernels, parallelism, quantization, and traffic shape can change real results, so the output is a hypothesis to validate with production-shaped traces.

The closest single-pool alternative is chunked prefill scheduling, which interleaves bounded prompt chunks with decode work. Compare that option before adding a second pool. Disaggregated prefill earns its complexity only when phase isolation and independent scaling beat the transfer and operating costs.

Measure prefill and decode as different services

Prefill processes many prompt tokens in parallel and usually wants high arithmetic throughput. Decode advances one token per active sequence and is commonly shaped by memory bandwidth, batching opportunities, and the latency cost of waiting for another scheduling step. That asymmetry is the architectural reason to examine prefill decode disaggregation.

Represent each phase with a small service record: worker count, effective tokens per second, fixed dispatch overhead, queue discipline, and utilization ceiling. Keep observed values separate from estimates. A capacity spreadsheet that silently treats peak vendor throughput as sustained service rate will recommend a topology that disappears under burst traffic.

The vLLM documentation describes disaggregated prefill as an experimental pattern and explicitly notes that it does not improve throughput by itself. That boundary is central. Separation can protect time per output token from prefill interference, but it introduces a KV cache transfer and another queue.

Record latency as a timeline, not one average: admission, prefill wait, prefill execution, transfer wait, KV movement, decode wait, first decode step, and remaining generation. The first thesis figure makes those lanes visible. A decision must explain which lane shrinks, which grows, and which uncertainty remains unmeasured.

Budget the KV cache transfer

A completed prompt leaves key and value tensors that decode needs. With disaggregated prefill, that state must move or become remotely accessible before generation can begin. Estimate bytes from layer count, token count, KV heads, head dimension, element size, and any implementation-specific layout factor. Treat the formula as a planning bound and verify it against the serving stack.

Transfer time includes serialization or registration, queueing, payload movement, synchronization, and receiver visibility. Link bandwidth alone is not the answer. Small transfers may be latency dominated; large contexts may be bandwidth dominated; concurrent transfers may contend with model or collective traffic. The lab therefore accepts measured effective bandwidth and fixed overhead as explicit inputs.

NVIDIA Dynamo documents disaggregated serving and routing, including the need to connect request routing with the prefill and decode topology. Use that implementation guidance to identify what must be measured in the selected stack, not as proof that one deployment will win.

Set a transfer budget before running scenarios. If time to first token may increase by at most 40 milliseconds, reserve only part of that envelope for KV movement and queueing. Disaggregated prefill fails the provisional gate when the conservative transfer estimate consumes the whole envelope, even if pool utilization looks attractive.

Model queues without inventing precision

A serving diagram becomes a decision tool only when it admits waiting. For each pool, calculate offered work per second from the request mix and phase demand, then divide by declared service capacity. Utilization near one is a warning, not a precise latency forecast. Tail behavior depends on arrival bursts, batching, scheduler policy, and service-time variance.

The planner uses a bounded utilization penalty to compare scenarios consistently. It labels the result modeled and refuses utilization at or above the configured ceiling. It does not print a fake p99. Instead, it shows headroom, the dominant queue, and the input that most affects the result.

The DistServe paper presents disaggregated prefill and decoding with goodput under service-level objectives. Its broader lesson is that phase placement should be evaluated against both time to first token and time per output token, not aggregate tokens per second alone.

Use the same discipline as the LLM serving goodput and SLO workflow: declare the objective, preserve the workload cohort, and count only work that satisfies it. If a disaggregated layout increases raw throughput while more requests miss first-token targets, the architecture did not improve the product contract.

Compare three topologies on one workload

Run aggregated, chunked, and disaggregated scenarios over the same request cohort. Aggregated serving keeps phase state local and has the fewest moving parts, but prefill can interfere with decode. Chunked scheduling keeps one pool while bounding prompt work per turn. Disaggregated prefill isolates phases and permits independent pool sizes, but pays transfer and routing costs.

Freeze the cohort: prompt-length buckets, expected outputs, arrival trace, service rates, worker inventory, objective thresholds, and network assumptions. Changing any of those between rows invalidates the comparison. The planner’s scenario matrix marks each value as observed, supplied, or derived.

The second figure plots first-token and inter-token budgets against operating complexity. Its purpose is not to crown a winner. It exposes the frontier: an aggregated plan may remain best for moderate prompts; chunking may protect decode without state movement; separation may help when long prompts are bursty and phase-specific capacity can stay busy.

Include LLM request preemption, recompute, and swap in the operational review. A new topology changes failure and overload choices. Decide what happens when a prefill worker finishes but no decode slot is available, a transfer stalls, or the destination loses state.

Route by phase state and live pressure

A disaggregated topology needs more than two worker labels. The router must know model identity, adapter or tenant scope, cache compatibility, pool health, queue age, destination capacity, and freshness of any placement evidence. A prefill completion is not useful if it points to a decode worker that cannot accept the state safely.

Prefer a two-step contract. Admission chooses whether the request belongs in the separated path at all. Placement then selects compatible prefill and decode capacity under live pressure. Short prompts or latency-sensitive requests can remain on an aggregated lane when transfer overhead would dominate. Large batch jobs can use the separated lane when queue isolation matters more.

The KV-aware routing guide shows why cache affinity must be balanced against live work and namespace safety. Reuse that receipt shape: record candidates, rejected workers, state scope, load evidence, chosen route, and the reason.

The third figure follows one request through admission, prefill, transfer, decode, and fallback. Every transition has a timeout and owner. That makes disaggregated prefill reviewable as a product system rather than a box-and-arrow optimization. The route receipt should be safe to retain without prompt text or customer identifiers.

Design failure and fallback before rollout

Separation multiplies partial states. Prefill can succeed while transfer fails. State can arrive after the destination admission window. A decode worker can restart after accepting metadata but before consuming the KV cache. Network degradation can preserve health checks while destroying the latency objective.

Define bounded outcomes for each edge. Retry transfer only when state identity and receiver capacity are still valid. Recompute prefill when the retry budget is exhausted and the request remains within its deadline. Fall back to an aggregated lane only when model and policy match. Cancel and release both sides when the client leaves.

Track orphaned cache bytes, transfer retries, recomputes, admission rejects, stale placements, and requests that cross the fallback boundary. These are not secondary infrastructure metrics; they explain user-visible pauses and capacity loss. Add a chaos fixture that delays transfer beyond its budget and prove the planner rejects the optimistic scenario.

Roll out by workload slice. Start with one model, bounded prompt buckets, and a small percentage of requests. Compare the same SLO cohorts before and after. Disaggregated prefill should be reversible at the router, with old and new paths observable at the same time. A feature flag without state cleanup and drain behavior is not a rollback plan.

Export a topology receipt and validate it

The decision receipt contains workload buckets, phase rates, worker counts, utilization ceilings, KV cache arithmetic, effective transfer assumptions, objective thresholds, scenario results, rejected topologies, fallback rules, and every input’s provenance. Export JSON for review and CSV for the scenario matrix.

Reparse both exports and recompute the headline values independently. Mutation fixtures remove transfer overhead, merge phase queues, substitute peak for effective bandwidth, ignore an overloaded pool, and report modeled latency as measured. The test oracle must reject each mutation.

Before production, replay representative traces on the actual software and network path. Measure time to first token, time per output token, goodput, transfer duration, queue age, GPU utilization, and failure counters. Keep the planning receipt beside the benchmark receipt so differences become evidence instead of surprise.

Disaggregated prefill is valuable when it makes phase interference and capacity ownership easier to control. It is expensive when it merely redraws one queue as three. The right result may be aggregated, chunked, or separated. Choose the smallest topology that passes the declared latency contract, preserve its truth boundary, and revisit it when traffic mix, serving software, model shape, or interconnect changes.

Runnable local artifact — The planner performs deterministic capacity arithmetic over supplied inputs; it is not a GPU, network, scheduler, or production benchmark.

Adjust a declared request cohort, effective phase rates, pool sizes, and transfer budget; compare feasible topologies and export JSON or CSV.