Sequence Packing Without Cross-Example Leakage

Sequence packing removes padding waste by placing several examples into one token buffer, but throughput is meaningless if one example can attend to another or its labels leak across a seam. This guide makes boundary correctness testable before a long fine-tuning run.

The outcome is a packing contract for token order, cumulative lengths, position resets, causal attention, label masks, and equivalence against an unpacked control.

The supporting vocabulary is dense token collation, segment-aware causal mask, example-local positions, loss-seam testing. Each term serves the same search intent: pack variable-length training examples without allowing attention or loss to cross example boundaries.

My position is that packing is a data transformation with semantic invariants, not a data-loader toggle. I do not trust a utilization chart until logits and loss agree with independently processed examples.

1. Token spans
2. Cumulative lengths
3. Attention seams
4. Loss mask

Sequence packing starts with semantic seams

Every example needs a start, end, position origin, attention segment, and loss policy carried beside its tokens. The Hugging Face packing with FlashAttention 2 explains padding-free concatenation with cumulative sequence lengths that preserve example boundaries. For this sequence packing decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Assign stable example IDs
• Emit cumulative sequence lengths
• Reset or transform position IDs
• Record supervised-token spans

The measurement I keep is a seam table reconstructing every packed example. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is tokens exist without a recoverable example boundary. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is every token maps to exactly one example and one loss decision. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Choose packing from the length distribution

A fixed recipe can waste space or reorder data badly when examples have a long tail. The TRL SFT Trainer documentation documents packing strategies and padding-free forward passes for supervised fine-tuning. For this sequence packing decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Plot token-length quantiles
• Compare first-fit and best-fit decreasing
• Measure dropped remainder
• Preserve shuffle semantics

The measurement I keep is non-padding tokens per allocated training token. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is packing improves density by distorting sample order. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is the packer must meet density and ordering budgets together. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Use boundary-aware attention

Causal order alone does not stop a later example from attending to tokens in an earlier example. The FlashAttention paper establishes the IO-aware exact-attention algorithm used by variable-length packed execution. For this sequence packing decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Build segment-aware causal masks
• Pass cumulative lengths to varlen kernels
• Test each backend explicitly
• Fail closed on unsupported masks

The measurement I keep is cross-segment attention probability and logit equivalence. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is the second example changes when the first example text changes. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is mutating a neighboring example must not change protected logits. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Option	Observed signal	Verdict
Padding	correct; token waste	control
Naive concat	fast; cross-example attention	reject
Boundary-aware pack	equivalent loss; dense tokens	ship

Handle position IDs as model behavior

Some models expect positions to reset at each example; others use packed offsets or specialized encodings. For this sequence packing decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Read the model's positional scheme
• Define reset behavior
• Test rotary inputs at seams
• Record maximum local and packed positions

The measurement I keep is position tensors compared with the unpacked control. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is a seam introduces shifted rotary phases. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is each example receives the position convention it would have alone. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Test loss-mask boundaries

Assistant-only or completion-only training can leak supervision when shifted labels cross a packed seam. For this sequence packing decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Mask the first label after each seam
• Preserve ignored prompt tokens
• Inspect shifted input-label pairs
• Compare token-level losses

The measurement I keep is per-token loss vectors before reduction. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is the last token of one example predicts the first token of the next. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is no target or reduction term may cross an example boundary. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Runnable artifact. Save this bounded check as sequence-packing-without-leakage.test.mjs and run node --test sequence-packing-without-leakage.test.mjs. Expected output: PASS: packed attention respects example seams.

import assert from "node:assert/strict";
import test from "node:test";
const attends=(segments,q,k)=>k<=q&&segments[q]===segments[k];
test("blocks cross-example attention",()=>{const s=[0,0,1,1];assert.equal(attends(s,2,1),false);assert.equal(attends(s,3,2),true);console.log("PASS: packed attention respects example seams");});

Prove packed and unpacked equivalence

A small deterministic batch can compare logits, loss, and gradients before throughput enters the conversation. For this sequence packing decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Disable stochastic layers
• Use identical token examples
• Run packed and separate forwards
• Compare inside dtype tolerance

The measurement I keep is maximum logit, loss, and gradient delta. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is aggregate loss matches while one example diverges. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is each example and supervised token must remain inside the declared tolerance. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

1. DefineDefine

   Write attention and label invariants.

2. PackPack

   Emit spans, offsets, and masks together.

3. CompareCompare

   Match unpacked logits and gradients.

4. TrainTrain

   Watch utilization and seam regressions.

Measure the speedup honestly

Token density is only useful when kernel selection, collation cost, and optimizer steps produce better end-to-end throughput. For this sequence packing decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Time collation separately
• Record real tokens per second
• Capture GPU utilization
• Keep effective batch tokens fixed

The measurement I keep is useful supervised tokens per wall-clock second. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is dense batches spend the gain in CPU packing. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is ship only when end-to-end useful-token throughput improves. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

Version the packing contract

Tokenizer, chat template, model backend, attention implementation, and loss policy all shape the seams. For this sequence packing decision, the useful move is to expose the hidden variable before optimizing the attractive output. That turns a technique into an operating rule another person can inspect.

Use four concrete actions:

• Hash tokenizer and template
• Record backend versions
• Save fixture packs
• Pin the equivalence tolerance

The measurement I keep is clean-run reproduction of seam tests. I record the input, configuration, observation window, and rejected control together. That bundle matters because a single favorable number cannot explain whether the method improved the system or merely moved cost into a quieter place.

The failure I deliberately provoke is a library update changes a mask convention silently. A check that never produces this bad case is too polite to prove its guardrail. I prefer the smallest counterexample that makes the break unmistakable, then I scale the experiment only after the mechanism is visible.

My decision rule is rerun boundary and equivalence checks after every dependency change. This is a proposed boundary from hands-on prototyping and systems review, not a claim about an undisclosed client deployment. A different workload, material, device, or visual goal can choose another answer, but it should publish the evidence that changed the boundary.

The useful version is bounded

Sequence packing is ready when the dense buffer behaves like the same examples processed separately. Utilization is the reward for preserving that equivalence, not a substitute for it.

Keep seam fixtures in CI and the unpacked control in the benchmark. They catch the quiet class of bugs that a smooth training-loss curve can hide.

Continue with LLM quantization quality budgets, model distillation, QLoRA memory receipts, QA notes that build trust. Those field notes deepen adjacent implementation choices without turning this page into several articles at once.