PyTorch CUDA Memory Snapshots: Find GPU Spikes

PyTorch CUDA memory snapshots turn a vague GPU spike into an allocator timeline you can inspect. This guide captures bounded history, separates live tensors from retained references and fragmentation, reconciles device memory outside PyTorch, and exports a redacted incident receipt.

Read PyTorch CUDA memory snapshots, not one peak

PyTorch CUDA memory snapshots answer a different question from a single “allocated” or “reserved” gauge. A peak tells you when capacity was high. The history explains which allocator segments existed, which blocks were active, which allocation sites survived, and what the allocator could not satisfy at the OOM event. Diagnosis needs that sequence.

Keep three ledgers separate. Device usage is the broadest total visible for the process. Reserved bytes are segments owned by the PyTorch caching allocator. Active bytes are blocks still associated with live allocations. The gap between reserved and active may contain reusable free blocks, unusable fragments, or bookkeeping. Device minus reserved suggests memory outside this allocator, not automatically a leak.

The official PyTorch CUDA memory snapshot guide documents history recording and the snapshot viewer, including an important boundary: allocator history cannot see every direct CUDA allocation. That is why this tutorial pairs a PyTorch memory snapshot with device reconciliation instead of naming a cause from one chart.

Begin incidents with a question, not a tweak: did the active live set grow, were references retained across steps, did free capacity become badly shaped, or did an external consumer take the missing bytes? The bounded workbench makes those four hypotheses explicit. For a planning baseline, compare the per-rank ownership model in the ZeRO-3 memory budget.

Model the caching allocator in three ledgers

The PyTorch caching allocator requests larger segments from CUDA, divides them into blocks, and reuses compatible free blocks to avoid synchronizing every allocation with the device. A block can be active, free, or unavailable for the request under review. Those states are useful only when their byte totals reconcile with the enclosing segment.

Represent every segment with a stable ID, address range, total size, stream or pool label, and ordered blocks. Represent each allocation event with logical time, action, requested size, granted block, allocation ID, and a redacted frame token. The incident oracle rejects overlapping blocks, blocks outside a segment, negative sizes, duplicate live allocation IDs, and totals that do not add up.

The CUDA memory management notes describe allocator controls and memory statistics. Treat those controls as experiments after classification. Changing a split threshold before proving fragmentation can trade one unexplained shape for another. Likewise, calling empty-cache operations may release unused reserved segments but cannot free tensors that remain reachable.

PyTorch CUDA memory snapshots are strongest when paired with a plain equation: device bytes = allocator-reserved bytes + external-or-unattributed bytes. Inside the allocator, reserved = active + reusable free + fragmented free + overhead. The labels are incident hypotheses, not universal internal categories. Preserve raw segment evidence so another engineer can recompute the classification without trusting the interface summary.

Capture a bounded and redacted incident window

Enable history before the suspected phase, then capture the shortest useful interval around warmup, forward, backward, optimizer, checkpoint, evaluation, or generation. PyTorch CUDA memory snapshots can become large and can expose stack paths, module names, or workload structure. Bound both event count and stack depth, save outside a public artifact path, and apply the organization’s incident-data policy.

The workbench uses at most 96 events, 16 segments, and 32 blocks per segment. Its fixture is synthetic: a baseline step, a retained activation, a split-heavy burst, a release, an external-memory rise, and one failed request. Logical ticks replace wall time so a replay produces the same ledger and hash. A production trace should preserve real timestamps separately.

Redaction happens before export. Convert frames to stable one-way tokens within the incident, strip user directories and request identifiers, and never export tensor values. The receipt states which fields were removed. A mutant deliberately copies a raw path; the export validator must kill it.

Capture overhead is part of the experiment. Reproduce on a controlled canary where possible, note allocator configuration, PyTorch and CUDA versions, device model, batch dimensions, and whether compilation or graph capture is active. PyTorch CUDA memory snapshots must travel with that cohort evidence. If checkpointing is under investigation, use the explicit compute-memory contracts in activation checkpointing tradeoffs rather than inferring a benefit from a lower final gauge.

Reconstruct the active-allocation timeline

Sort events by their recorded sequence, not by the visual order of stack labels. Apply allocation only to a known free block, apply free only to the matching live allocation, and retain a running active-byte total after every event. The first diagnostic chart should show active bytes, reserved bytes, device bytes, and failed request size on aligned lanes.

A rising active curve that returns after each phase is a peak-capacity problem. A staircase that never returns suggests retained references or intentionally persistent state. A stable active curve with growing reserved bytes may reflect allocator reuse or fragmentation. A device curve that rises while reserved stays flat points outside the PyTorch caching allocator. None of these shapes alone names the source line; frames and phase markers narrow it.

The synthetic PyTorch memory snapshot marks phase boundaries and lets the independent oracle rebuild lifetimes from raw events. It compares every derived interval against allocation and free ticks, checks the peak value, and rejects an event that frees the wrong ID. The UI cannot certify its own timeline.

Annotate semantic phases in the workload itself. “Step 17” is less useful than “backward after attention” or “optimizer state initialization.” Correlate shape changes with batch, sequence, image, or generation dimensions. For low-rank training, QLoRA fine-tuning memory provides a separate ledger for quantized base weights, adapters, gradients, optimizer state, and activations.

Reconcile memory the snapshot cannot see

PyTorch CUDA memory snapshots describe memory managed by the allocator that produced the history. They may not account for NCCL, another framework, custom CUDA extensions, driver workspaces, or allocations made directly through CUDA APIs. The incident receipt therefore records device-used bytes beside allocator-reserved bytes at the same phase boundary.

Compute the residual as device used minus allocator reserved, but label it external or unattributed. Sampling skew, asynchronous work, and accounting differences can make small residuals noisy. Require a meaningful, sustained delta before classifying hidden memory. The workbench’s external fixture raises the device ledger while allocator totals stay fixed; a mutant hides that lane and is rejected by the independent classifier.

The PyTorch engineering post Understanding GPU Memory demonstrates how allocator history and reference-cycle evidence can explain memory behavior over time. The transferable lesson is to align evidence sources, not to assume every incident shares one cause.

If the residual is material, isolate libraries and phases. PyTorch CUDA memory snapshots should be paired with one broader device ledger at the same boundary. Disable one optional communication or extension path in a controlled reproduction, compare before and after, and record the experimental difference. Do not claim a leak merely because memory remains cached or because a library owns a persistent workspace. The correct conclusion may be “unattributed under current instrumentation,” with a smaller follow-up probe.

Classify live set, retention, fragmentation, or external use

CUDA OOM debugging becomes actionable when classification rules are explicit. Choose live-set pressure when active bytes plus the failed request exceed available device capacity even under ideal packing. Choose retention when equivalent phase boundaries show a monotonic increase in still-live allocations and frames point to repeated ownership. Choose GPU memory fragmentation when total free allocator bytes could satisfy the request but no compatible contiguous block can.

Choose external pressure when allocator-reserved bytes remain stable while reconciled device usage grows materially. More than one condition can be true, so the receipt records primary and contributing diagnoses. It also records rejected explanations. A peak by itself is not a cause, and “reserved is high” is not proof of fragmentation.

The independent oracle recomputes maximum free block, total free bytes, active growth across matched boundaries, and external residual. Mutation fixtures delete active bytes, merge separate holes, hide external memory, and substitute the peak label for the decision rule. Each altered trace must disagree with the oracle.

Compilation can legitimately change allocation timing and persistence. When a spike appears after enabling compilation, compare the allocator evidence with torch.compile graph breaks so graph fragmentation and memory fragmentation do not become one vague story. PyTorch CUDA memory snapshots supply the allocator half; compiler traces supply the execution half.

Apply the smallest evidence-backed fix

Fix the classified constraint and keep everything else stable. For a true live-set overflow, reduce batch or sequence dimensions, change activation strategy, shard state, shorten generation, or schedule competing work separately. For retention, release the owning references, detach diagnostic tensors, bound caches, and verify the matched phase returns to baseline.

For fragmentation, first prove a failed request fits total free bytes but not the largest compatible block. Then test allocator configuration or allocation-shape regularity on the same reproduction. Record both successful allocations and throughput effects. Avoid universal configuration recipes: allocator behavior depends on versions, shapes, streams, and workload order.

For external pressure, move the investigation to the owning library or direct CUDA path. Measure at the same synchronization boundary and confirm that removing or rescheduling that consumer reduces the residual. If attribution remains incomplete, reserve headroom rather than inventing precision.

Every remediation needs a before-and-after contract: identical fixture dimensions, allocator configuration, software cohort, warmup, phase markers, and capture window. Compare peak active, peak reserved, largest free block at failure, external residual, and whether the original request succeeds. The PyTorch memory snapshot receipt marks model-derived fields separately from directly observed fields, so a cleaner chart cannot masquerade as a proven capacity win.

Export the incident receipt and rerun the proof

A useful receipt contains the capture boundary, redacted environment, phase dictionary, allocator totals, device reconciliation, failed request, classification inputs, primary and contributing diagnoses, rejected hypotheses, remediation, and before-and-after checks. It includes a digest of canonical evidence but excludes raw stack paths and tensor data.

Export JSON for machine comparison and CSV for the event ledger. Reparse both outputs, verify stable column order, assert every exported allocation ID exists in the source fixture, and ensure redacted frame tokens match a strict pattern. The browser workbench performs these checks locally and makes no network request. Its result remains a synthetic demonstration, not a statement about a real GPU.

Before closing the incident, rerun the exact capture after the smallest fix. Require the failed request to complete or the target headroom to return, while checking throughput and correctness separately. If the diagnosis changes under identical evidence, preserve both receipts and investigate the oracle or environment drift.

PyTorch CUDA memory snapshots are valuable because they replace a single alarming number with reconstructable allocator history. Their discipline is equally important: bound capture, redact sensitive frames, reconcile what the allocator cannot see, name uncertainty, and retain raw evidence. Schedule the January review when APIs, allocator semantics, or incident fixtures change—not merely to refresh the date.

Runnable local artifact — The workbench models a deterministic synthetic allocator trace; it is not a GPU benchmark and does not inspect a real process.

Replay a frozen allocator incident, compare active, reserved, fragmented, and external bytes, classify the failure independently, and export a redacted receipt.