CUDA Graphs for Stable LLM Serving

CUDA graphs can remove repeated CPU launch overhead from LLM serving, but capture only helps when shapes, allocations, and control flow remain inside the graph's contract. This guide maps capture buckets, replay paths, fallbacks, and memory cost before promising a latency win.

The result is a serving policy that says which requests replay, which run eagerly, and how to detect when the captured fast path has stopped representing real traffic.

The supporting vocabulary is captured replay coverage, kernel launch overhead, static shapes, LLM inference latency. Each term serves the same search intent: use CUDA graph capture for LLM serving without hiding dynamic-shape fallbacks or memory costs.

My position is that graph coverage is a product metric. A fast captured benchmark is irrelevant if most customer requests miss its shape buckets.

1. Warmup
2. Capture buckets
3. Replay path
4. Eager fallback

CUDA graphs start with launch evidence

Graph capture should answer measured CPU launch gaps rather than decorate an already GPU-bound trace. The CUDA Programming Guide defines graph construction, instantiation, and replay semantics. In this CUDA graphs method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Profile eager execution
• Mark CPU idle gaps
• Separate prefill and decode
• Keep a no-graph control

The useful measurement is launch overhead as a share of step time. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is capture changes nothing because kernels dominate. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is proceed only when launch cost is material. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives CUDA graphs a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Map dynamic shapes

Batch size, sequence length, multimodal items, and speculative slots decide whether a replay matches. The PyTorch CUDA graphs article explains warmup, static memory addresses, and replay in PyTorch. In this CUDA graphs method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Log shape tuples
• Build a frequency histogram
• Include burst periods
• Mark incompatible control flow

The useful measurement is traffic coverage by candidate bucket. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is the benchmark uses a shape customers rarely send. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is capture buckets must explain a declared traffic share. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives CUDA graphs a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Choose a small capture set

Every graph consumes memory and warmup time, so exact-shape completeness is usually the wrong goal. The vLLM engine arguments documents configurable CUDA graph capture sizes in an LLM serving engine. In this CUDA graphs method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Start from frequent shapes
• Estimate padding waste
• Measure graph memory
• Add buckets by marginal coverage

The useful measurement is coverage gained per memory unit. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is capture proliferation steals KV capacity. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is stop when the next bucket costs more capacity than latency it saves. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives CUDA graphs a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Option	Observed signal	Verdict
One max graph	memory high; padding waste	reject
Many exact graphs	capture cost explodes	reject
Traffic buckets	coverage and memory balance	ship

Respect capture constraints

Allocations, addresses, and side effects need stable behavior across warmup and replay. In this CUDA graphs method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Warm on a side stream
• Reuse static buffers
• Remove capture-unsafe calls
• Assert address stability

The useful measurement is successful replay with identical outputs. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is a hidden allocation or sync breaks replay. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is fallback must be safe whenever capture preconditions fail. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives CUDA graphs a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Test bucket selection

A deterministic selector should choose the smallest adequate graph and reject oversize requests. In this CUDA graphs method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Sort bucket sizes
• Select a ceiling bucket
• Return eager beyond maximum
• Test exact boundaries

The useful measurement is selected bucket for known shapes. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is a request is truncated or assigned below its shape. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is selection must never under-allocate. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives CUDA graphs a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Runnable artifact. Save this bounded check as cuda-graphs-llm-serving.test.mjs and run node --test cuda-graphs-llm-serving.test.mjs. Expected output: PASS: selector never under-allocates.

import assert from "node:assert/strict";
import test from "node:test";
const bucket=(n,sizes)=>sizes.find(x=>x>=n)??null;
test("ceil bucket",()=>{assert.equal(bucket(9,[1,8,16,32]),16);assert.equal(bucket(40,[1,8,16,32]),null);console.log("PASS: selector never under-allocates");});

Measure replay correctness

Speed cannot override token equality, cache position, sampling state, or cancellation behavior. In this CUDA graphs method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Compare logits on fixed inputs
• Compare generated tokens
• Exercise cancellation
• Reset mutable buffers

The useful measurement is eager-versus-replay output equivalence. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is replay leaks state between requests. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is every captured bucket passes paired semantic checks. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives CUDA graphs a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

1. ProfileProfile

   Measure launch gaps and traffic shapes.

2. BucketBucket

   Choose a small capture set from demand.

3. ReplayReplay

   Verify outputs and allocation stability.

4. WatchWatch

   Track coverage, fallbacks, and tails.

Count the memory bill

Captured workspaces and static buffers compete with weights, KV cache, and request capacity. In this CUDA graphs method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Trace allocation before capture
• Trace after every bucket
• Measure KV capacity change
• Include fragmentation

The useful measurement is bytes per graph and lost concurrency. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is lower step latency reduces total served requests. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is graph memory may not violate the concurrency floor. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives CUDA graphs a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

Monitor fast-path coverage

Traffic changes after release, so replay rate and fallback tails need continuing visibility. In this CUDA graphs method, the important move is to make the hidden variable visible before optimizing the attractive output.

Use four concrete actions:

• Emit bucket labels
• Count eager fallbacks
• Slice latency by path
• Alert on coverage drift

The useful measurement is qualified replay share over time. Record the input, configuration, observation window, and rejected control together. That bundle makes the result debuggable: another reviewer can tell whether a change improved the system or merely moved cost into a quieter part of the experience.

The failure to provoke is p95 worsens while graph benchmark remains green. A test that never produces that failure is too polite; it cannot show that the guardrail works. Design the smallest counterexample first, then scale the experiment only after the bad case is unmistakable.

My decision rule is revisit buckets when coverage or shapes move materially. This is a proposed operating boundary, not a claim about an undisclosed client system. It gives CUDA graphs a defensible stopping point while leaving room for a different workload, visual goal, or device constraint to choose another answer.

The useful version is bounded

Capture the traffic you actually serve, not a perfect static demo. Coverage, fallback behavior, and memory cost belong beside the latency number.

A good CUDA graph policy is explicit about the eager path. Dynamic requests are not errors; they are part of the system the fast path must coexist with.

Continue with continuous batching, tensor parallelism, activation checkpointing, QA receipts after deploy. Those field notes deepen adjacent implementation choices without turning this page into several articles at once.