Why can extending a model's context window regress its short-prompt quality?

## The tax is real and it is measurable Teams that extend a context window usually evaluate the *new* capability — can it answer questions about a 100K-token document — and forget to re-evaluate the *old* one. That is where the surprise lives. Post-hoc extension is not additive; it modifies the model globally, and the modification lands on short prompts too. ## Mechanism one: rescaling is not conditional Rotary position encoding gives each position a set of rotation angles, some fast-rotating (carrying near-neighbour relationships) and some slow (carrying coarse document-scale position). A static rescale divides positions by a fixed factor chosen for the target length. That factor applies to *every* request. So a 400-token prompt, which used to occupy positions 0-400 with full angular separation between adjacent tokens, now occupies a compressed sliver of the same range. The model's learned sensitivity to "this token is three back" versus "this token is five back" was calibrated at the original spacing. Compress it and the discrimination gets noisier. On tasks that depend on tight local structure — code, tables, precise instruction following, exact quotation — that shows up as errors that were not there before. The standard fix is **dynamic scaling**: derive the scale factor from the actual sequence length at inference time, so a short prompt is served at (or near) a factor of 1 and only genuinely long inputs are squeezed. Frequency-aware recipes help for a different reason — they concentrate the compression on the slow dimensions and leave the fast, local ones close to intact, so the local-resolution loss is smaller for the same nominal window. ## Mechanism two: the continued-training data shift The long-sequence training stage necessarily runs on a data mix that looks nothing like production traffic for a short-prompt product: whole repositories, book-length text, multi-day build logs, packed multi-document samples. Any substantial training run on a skewed mix moves the model. Instruction-following crispness, formatting habits, refusal calibration and short-answer accuracy can all drift, and drift is not always in the direction you would guess from the loss curve. The mitigation is unglamorous: **replay**. Mix a meaningful proportion of short, instruction-shaped, post-training-style data back into the long-context stage so the short behaviour keeps getting gradient signal. This is standard practice and it is why the stage is treated as a full data-mixture design problem rather than "train on some long documents". ## How you actually detect it The discipline is simple and frequently skipped: 1. **Freeze a short-context eval suite before you extend anything.** Your own task evals, not only public benchmarks — the short building-code question-answering set your product actually serves, sliced by the segments you care about. 2. **Run it on the base checkpoint** to establish the number you are defending. 3. **Run the identical suite on the extended checkpoint**, at identical decoding settings and identical prompts. Changing the sampling configuration at the same time as the checkpoint makes the comparison worthless. 4. **Slice by input length.** An aggregate score can hide the shape of the damage: often the loss is concentrated in the shortest bucket, which is precisely the traffic that dominates production. 5. **Look beyond accuracy.** Format adherence, output length distribution, and instruction compliance regress in ways an accuracy score does not surface. A regression of one to three points on short tasks is a common, expected outcome. The judgment call is whether that is worth the new long-context ability. ## The two-checkpoint decision When the tax is unacceptable and the mitigations do not close it, the honest option is to **stop trying to have one model**. Keep the original checkpoint for the short-prompt path and route only genuinely long requests to the extended one. The costs are real — two models to host, two to evaluate, two to keep in sync through future updates, plus a routing rule that will occasionally send a request to the wrong one — so it is worth doing only when the short path is high-volume and the regression is material. The alternative many teams take instead is to avoid post-hoc extension entirely and adopt a model that was built long, where the long-context capability came from the training pipeline and the architecture rather than from stretching a shorter model afterwards. ## What to say in an interview The strong answer names both mechanisms (global rescaling, data shift), names the detection method (a frozen pre-extension short-context suite, rerun under identical settings, sliced by length), and treats the outcome as a tradeoff to be measured rather than a bug to be denied. Claiming extension is free is the single clearest tell that someone has only read about it.