Why does packing several SFT examples into one sequence risk cross-example contamination?

## The efficiency problem packing solves SFT datasets are usually very uneven in length. A batch is a rectangle, so every sequence is padded to the longest one in it, and padding positions do work that is thrown away. With a 4,096-token window and a median example of 400 tokens, an unpacked batch can be 80–90% padding — you are paying for compute that contributes nothing to any gradient. **Sequence packing** concatenates examples end to end into a single full-length sequence, so each training token is a real token. On short-example datasets this is often a multiple-times speedup for identical learning, which is why it is default hygiene in production SFT rather than an exotic optimisation. ## What breaks if you just concatenate A causal transformer lets every position attend to all earlier positions in its sequence. Once you have glued unrelated examples together, "earlier positions" includes other people's examples. Two things go wrong. **Cross-example attention.** The answer tokens of the fourth packed example can attend to the transcript and answer of the first. The gradient therefore rewards using that context. You have quietly changed the task from "answer given this prompt" to "answer given this prompt and some unrelated text that happens to precede it." The damage is subtle: the model learns weak spurious correlations, and it becomes sensitive to whatever text precedes a request at inference — where, of course, no such preceding pack exists. **Position drift.** Positional information is assigned by offset within the sequence. Under naive packing, the same short example is at positions 0–400 when it lands first in a pack and at positions 3,100–3,500 when it lands last. The model sees the same content at wildly different positional encodings, which adds noise and, with rotary-style encodings, can push short examples into positional regimes they never occupy at inference. ## The correct implementation Two mechanisms, applied together: **Block-diagonal (document-boundary) attention.** Instead of a single causal mask over the whole packed sequence, build a mask that is causal *within* each example and zero across examples. Every token attends only to earlier tokens of its own example. Efficient attention kernels support this by taking the cumulative sequence lengths of the packed examples, so it costs essentially nothing — in fact it does less work than the dense mask. **Position-id resets.** Restart the position counter at each example boundary, so each packed example sees the same positions it would have seen alone. And, independently, the label mask still applies per example: each packed example contributes loss only on its own completion span, so a pack of five yields five separate supervised spans separated by ignored prompt regions. ## Second-order effects worth naming **Loss weighting shifts.** Averaging cross-entropy over all supervised tokens in a pack means examples with long completions contribute more than examples with short ones. Unpacked batches, averaged per example, weight examples equally. Neither is wrong, but the effective weighting changes when you turn packing on, so a like-for-like comparison against an unpacked baseline is not exactly like-for-like. **Truncation at the boundary.** A naive packer that fills to exactly the window length will cut the last example mid-completion, training a truncated target with no stop token. Packers should either move a straddling example to the next pack or, if splitting, do so knowingly. **Ordering artefacts.** Packing examples in dataset order groups similar examples into the same pack. Even with correct masking this correlates the examples inside a gradient step; shuffling before packing avoids it. **Detection.** Contamination from bad packing is hard to see in the loss curve — it typically shows as slightly *better* training loss, since extra context is extra information. The practical check is a controlled comparison: a short run with packing off, or with masking explicitly enabled versus disabled, evaluated on held-out prompts presented alone. If quality on single, isolated prompts is worse than the training curve implies, cross-example leakage is a prime suspect. ## The judgement call Packing is close to free when the stack supports boundary-aware attention, and it is a real correctness risk when it does not. If you are running on tooling where you cannot confirm that document boundaries are respected, the honest options are to leave packing off and accept the padding cost, or to bucket examples by length so padding waste shrinks without any concatenation. The one thing you should not do is concatenate and hope, because the failure is silent and the loss curve will actively reassure you.