torch.compile Graph Breaks: Find the Real Cost

torch.compile graph breaks do not automatically mean your model recompiles, and a clean single graph can still churn through specializations. This tutorial separates eager islands, guard misses, cache behavior, warmup, and steady calls so you fix only the cost your workload proves.

torch.compile graph breaks are not recompiles

A graph break is a boundary in one captured execution: TorchDynamo stops a compiled region, runs something eagerly, and may resume capture afterward. A recompile is a different event: a later call misses the guards for an existing compiled graph and produces another specialization. One request can contain two breaks without recompiling on the next call; another can stay in one graph yet compile repeatedly as shapes or Python values change.

That distinction changes the first question. Do not ask how many warning lines appeared. Ask how many compiled regions ran, how many eager islands crossed the hot path, which guards missed, and how often new variants were compiled. The PyTorch compiler programming model describes graph breaks and guards as separate mechanisms. My useful unit is therefore a call ledger, not a single global counter.

The ledger keeps five columns: call identity, graph identity, break reason, cache outcome, and measured phase. If the same graph IDs return with cache hits, fragmentation may be stable. If graph IDs multiply for equivalent work, PyTorch recompilation is the pressure. That framing prevents an aesthetically pleasing “zero breaks” project from replacing a measured performance problem.

Capture an eager and compiled baseline

Begin with a representative fixture, not a toy tensor whose control flow never resembles production. Freeze the model revision, device, dtype, input families, mode, and backend. Record eager latency first, then wrap the same callable with torch.compile and keep compilation warmup separate from steady calls. Synchronize the device around timing when the backend is asynchronous.

For torch.compile graph breaks, I keep three buckets. Warmup contains first capture and code generation. Steady hit contains calls that reuse an existing specialization. Changed input deliberately exercises dynamic shapes or Python values. Each bucket retains the same number of measured calls for baseline and candidate, so averages cannot hide unequal work.

The fixture should also preserve correctness evidence. Compare eager and compiled outputs with tolerances appropriate to the dtype; record exceptions and mutations of state; and keep seeds fixed where randomness is intentional. The goal is not to prove every possible model input. It is to create a narrow, reproducible trace whose timing and graph-accounting claims can be independently replayed before touching the model.

Read break locations and reasons together

A break count without its location is nearly useless. Turn on the smallest diagnostic surface that answers the question, usually TORCH_LOGS=graph_breaks,recompiles, then retain the operator or source location, reason code, and surrounding model phase. The compiler observability guide documents the logging and inspection tools; it does not make every emitted warning equally expensive.

Group identical locations before prioritizing them. A break inside a once-per-request tokenizer bridge has a different cost from the same break inside every decoder layer. Mark whether eager work synchronizes the accelerator, copies data, mutates Python state, or merely performs a cheap scalar decision. For each location, attach call frequency and wall time rather than inferring impact from stack-trace length.

This is where torch.compile graph breaks become an engineering problem instead of a warning-cleanup exercise. The receipt can say: “break B3 occurs twice per call, creates a 0.14 ms eager island, and remains stable across 80 steady calls.” That statement is bounded, falsifiable, and much more actionable than “TorchDynamo could not trace this function.”

Separate fragmentation from guard churn

Graph fragmentation describes how one call is partitioned. Guard churn describes how the compiled cache changes across calls. Plot them on separate axes. Region count and eager-island duration belong on the fragmentation axis; cache misses, new graph variants, and compile duration belong on the churn axis. Only then can dynamic shapes be evaluated honestly.

Suppose a service runs four stable regions with two cheap breaks and one compiled variant for every supported batch size. Flattening those regions might save little. Suppose another runs one region but recompiles for each sequence length because a Python integer was specialized. That path looks clean in a single trace while warm requests still pay compiler cost. The second problem can dominate despite having fewer breaks.

I use the CUDA Graphs serving guide as the adjacent systems check: capture safety, stable addresses, and replay constraints can amplify a seemingly small shape-policy decision. I also compare phase boundaries with Server-Timing for AI latency so compiler diagnostics do not swallow queueing, transfer, or downstream generation time. Each tool keeps ownership of a different layer of the story.

Choose a rewrite, custom operator, or eager boundary

A fix earns its place only when it targets the measured cause. Rewrite trace-hostile Python when equivalent tensor operations preserve semantics and reduce an expensive island. Use a custom operator when the operation has a stable contract that the compiler may treat opaquely, with explicit fake or meta behavior where required. Keep an eager boundary when the code is rare, side-effectful, data-dependent in an essential way, or cheaper than the complexity of making it compilable.

For torch.compile graph breaks, “fullgraph succeeds” is not automatically the objective. A forced rewrite can expand specialization surfaces, increase code size, or move work into a custom kernel that is harder to validate. Record the semantic risk, expected latency change, and rollback boundary for every candidate. Then run the same call families through the baseline and candidate.

The decision also belongs beside memory choices. Activation checkpointing trade-offs can change recomputation and phase structure even when compiler regions are unchanged. Treat compiler, memory, and serving policy as related ledgers rather than collapsing them into one magic optimization switch.

Measure warmup and steady calls separately

A compile experiment needs an explicit chronology. I run an eager control, a cold compiled call, a fixed number of warm cache hits, and a bounded family of changed inputs. Cold and warm distributions are reported separately. If deployment creates fresh workers frequently, warmup belongs in capacity planning; if workers live for days, steady behavior may carry more weight. Neither number should disappear into one mean.

The PyTorch 2 technical report explains the TorchDynamo, AOTAutograd, and backend compilation stack that contributes to this path (PyTorch 2 paper). A trace receipt should not attribute every cold millisecond to a graph break. It should identify observed compile events, cache hits, eager-island time, and total call time, while naming unmeasured backend work as a limitation.

Keep equal sample counts. Compare medians and a tail percentile only after confirming equivalent inputs and outputs. For workload selection, the reasoning in vLLM vs SGLang for agent workloads is a useful parallel: the representative request mix matters more than a detached peak number. The same discipline applies to torch.compile graph breaks.

Kill misleading trace-accounting mutations

The lab accompanying this article imports a bounded JSON trace and computes a baseline/candidate ledger. Its negative fixtures are not labels pasted onto expected failures. They alter the production calculation, then an independent oracle looks for consequences.

Five mutations matter. Conflate breaks and recompiles increments compile count at every break. Drop warmup removes cold calls before comparing total cost. Count cache hits as compiles turns reuse into churn. Average unequal calls divides baseline and candidate totals by different hidden sample counts. Hide eager islands zeros observed eager duration while retaining region transitions. Each mutant must change the receipt hash and trigger a semantic failure code.

This mutation set protects the core thesis: torch.compile graph breaks and recompiles answer different questions. It also guards the arithmetic around them. A green baseline is insufficient if the verifier cannot reject plausible bad bookkeeping. The exported JSON and CSV include the input trace hash, call counts, phase totals, observed graph IDs, failure codes, and the declared limits, so another person can reproduce the result without trusting the interface.

Keep a compilation decision receipt

The final artifact is a decision record, not a victory screenshot. It names the model and environment, trace fixture, warmup policy, call families, correctness tolerance, graph and break summaries, guard-miss evidence, timing distributions, chosen intervention, and rollback trigger. It also records what the trace does not establish: universal speedup, unseen input stability, backend portability, or production capacity.

For each torch.compile graph breaks investigation, I keep the before and after receipts together. If a later PyTorch release changes a guard or decomposition, the comparison can be rerun against the same sanitized trace. If the candidate saves steady time but doubles cold compilation, the owner can make that trade with the worker-lifetime distribution in view.

The smallest safe decision may be to fix one repeated scalar extraction, mark one honest eager boundary, or change a shape policy. The receipt keeps that restraint visible. Compiler work is successful when the deployed workload becomes cheaper and remains correct—not when every diagnostic line disappears.

Runnable local artifact — The lab audits a sanitized bounded trace fixture and arithmetic only; it does not run PyTorch, profile a GPU, predict production speedup, or generalize beyond the imported calls.

Import or select a frozen trace; retain at most 64 calls and 16 events per call; compare equal baseline/candidate phases; kill five production-path mutations; export canonical JSON or CSV.