PagedAttention Fragmentation You Can Measure

PagedAttention fragmentation is smaller than the waste from reserving every request's maximum context, but it is not zero. This guide separates tail slack, free-block scatter, metadata, copy-on-write sharing, and allocation churn so a block-size choice has evidence behind it.

The goal is not to prove paging works in the abstract. It is to find the block size that preserves useful batch capacity under the prompt lengths, output lengths, cancellations, and forks your server actually sees.

The supporting vocabulary is KV paging ledger, free-page count, cache allocation churn, inter-token latency. Each term serves the same search intent: measure residual KV-cache waste and choose a block size from real sequence churn rather than a peak-memory snapshot.

My position is that memory efficiency should be read as admitted useful tokens under a latency target. A low allocator percentage can still be a bad result if small blocks add lookup overhead or churn prevents stable batching.

1. Token occupancy
2. Tail slack
3. Free blocks
4. Batch capacity

PagedAttention fragmentation needs a byte ledger

Allocated GPU memory should be divided into payload, unused block tails, tables, allocator reserve, and non-cache workspace. The PagedAttention paper introduces paged KV-cache management to reduce fragmentation and redundant duplication in LLM serving. For this PagedAttention fragmentation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Sample live sequence lengths
• Record physical block ownership
• Count final-block slack
• Separate reserved from resident bytes

The measurement I keep is useful KV bytes divided by the complete cache pool. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is allocator reserve is reported as useful cache. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is every byte category reconciles to the observed pool total. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Replay sequence churn, not static lengths

Fragmentation appears as requests grow, fork, cancel, and release pages at different times. The Hugging Face paged attention documentation documents block tables, cache sequence lengths, and non-contiguous physical cache locations. For this PagedAttention fragmentation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Preserve arrival ordering
• Replay decode growth
• Include beam or prefix sharing
• Keep cancellation bursts

The measurement I keep is occupancy and alloc/free events over time. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is a static packed batch hides churn-induced pressure. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is the chosen size must survive the production-shaped lifecycle trace. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Sweep block size in isolation

Block size trades last-page slack against block-table depth, kernel behavior, and allocation frequency. The vAttention paper analyzes physical-memory fragmentation and the programming overhead introduced by non-contiguous PagedAttention layouts. For this PagedAttention fragmentation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Choose a small logarithmic sweep
• Hold model and traffic fixed
• Warm kernels consistently
• Repeat cold and steady phases

The measurement I keep is qualified throughput, slack, tables, and tail latency. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is the smallest block wins memory but loses latency. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is choose the knee that admits the most useful work inside the SLO. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Option	Observed signal	Verdict
Large blocks	low metadata; high tail slack	reject
Tiny blocks	low slack; lookup pressure	inspect
Workload knee	stable qualified batch	ship

Separate internal and external pressure

Tail slack is internal fragmentation; scattered free blocks are not necessarily unusable when any free page can satisfy the next request. For this PagedAttention fragmentation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Label final and full blocks
• Track the number of free pages
• Record allocation failures
• Inspect largest request admission

The measurement I keep is slack bytes plus failed admissions at equal free capacity. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is a dashboard calls every non-contiguous page wasted. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is only bytes that cannot serve declared work count as operational waste. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Test the tail-slack bound

For a sequence of n tokens and a block size b, unused tail capacity should stay below one block per live sequence. For this PagedAttention fragmentation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Calculate blocks with ceiling division
• Derive used and slack tokens
• Test exact boundaries
• Test one token past a boundary

The measurement I keep is maximum slack per live sequence. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is a boundary length allocates an extra empty block. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is tail slack remains between zero and block size minus one. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Runnable artifact. Save this bounded check as pagedattention-fragmentation.test.mjs and run node --test pagedattention-fragmentation.test.mjs. Expected output: PASS: tail slack stays below one block.

import assert from "node:assert/strict";
import test from "node:test";
const slack=(tokens,block)=>Math.ceil(tokens/block)*block-tokens;
test("bounds tail slack",()=>{for(const n of [1,15,16,17,63])assert.ok(slack(n,16)>=0&&slack(n,16)<16);console.log("PASS: tail slack stays below one block");});

Account for copy-on-write sharing

Shared prefixes reduce duplication, but a fork that diverges can create copy cost and temporarily raise block demand. For this PagedAttention fragmentation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Mark reference counts
• Trace shared-to-private transitions
• Measure copied tail bytes
• Replay simultaneous divergence

The measurement I keep is net bytes saved after copy events. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is sharing looks free until many children diverge together. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is prefix sharing ships only with a bounded divergence burst. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

1. ReplayReplay

   Preserve lengths, forks, and cancellations.

2. AccountAccount

   Separate payload, slack, tables, and free pages.

3. SweepSweep

   Change only block size and pool policy.

4. GateGate

   Choose admitted tokens inside latency SLOs.

Read latency beside occupancy

More admitted sequences help throughput only if block lookups and gathers do not stretch visible token cadence. For this PagedAttention fragmentation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Capture cadence between output tokens
• Profile attention kernels
• Correlate table depth with steps
• Retain a contiguous control

The measurement I keep is completed tokens inside TTFT and ITL budgets. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is high occupancy produces slower streams. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is optimize qualified batch capacity rather than raw resident sequences. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Version the memory recipe

Block size, kernel, model shape, dtype, sharing policy, and traffic distribution form one coupled result. For this PagedAttention fragmentation decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Hash engine and kernel versions
• Record KV dtype and head shape
• Save replay trace hash
• Publish the rejected controls

The measurement I keep is clean-process reproduction of the byte ledger. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is an engine upgrade silently changes allocation behavior. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is rerun the sweep after any cache-layout or workload-shape change. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

The useful version is bounded

PagedAttention fragmentation is a workload measurement, not a universal percentage. The defensible block size is the one that turns more cache bytes into useful, timely tokens.

Keep the byte ledger and churn trace next to the configuration. They explain why the choice exists when a model, kernel, or traffic mix changes.

Continue with KV-cache optimization, chunked prefill, continuous batching, memory-mapped model loading. Those field notes deepen adjacent implementation choices without turning this page into several articles at once.