When do you enable YaRN rope_scaling on a self-hosted Qwen model, and how?

## What the native window is A checkpoint's `config.json` carries `max_position_embeddings`, the sequence length the model was trained and validated at. Qwen releases have varied here: many Qwen2.5 and Qwen3 dense checkpoints ship a 32,768-token native window, while some later Qwen3 releases ship a much larger native length outright. Always read the checkpoint's own config and model card rather than assuming a family-wide number — this is exactly the kind of fact that rots between generations. Past that length, positional encoding is extrapolating into territory the model never saw, and quality collapses rather than degrading gracefully. ## What YaRN does Qwen uses rotary position embeddings, where position enters attention as a rotation whose frequency varies by dimension. YaRN is a length-extension method that rescales those frequencies non-uniformly — leaving the high-frequency dimensions that encode local ordering largely intact while compressing the low-frequency ones that encode long-range position, so positions beyond the training length map into a range the model can still interpret. The practical effect is that a model trained at 32k remains coherent out to roughly 4× that. For the person operating a server, the important properties are: it is a configuration change, not a retrain; it costs nothing at load time; and it is applied uniformly to every request. ## How you turn it on The canonical form is a block in `config.json`: "rope_scaling": { "rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768 } `factor` is the multiplier over the original length — 4.0 takes 32,768 to 131,072 — and `original_max_position_embeddings` must be the model's real native length, because that is the boundary the method scales relative to. Editing the file in the downloaded model directory works, but it mutates a cached artifact, so the cleaner route is to pass the same JSON as a config override on the serving command line (vLLM accepts Hugging Face config overrides for exactly this; older builds exposed a dedicated rope-scaling argument). Either way you must also raise `--max-model-len` to the extended length, or the server will still refuse anything longer than the original window. ## Why it is off by default This is *static* YaRN: once configured, the rescaled frequencies apply to a 200-token prompt exactly as they do to a 100,000-token one. Qwen's documentation is explicit that this degrades performance on short inputs, so a server configured for 128k because "someone might paste a book" is quietly worse at the 99% of traffic that is a few thousand tokens. The engineering discipline is to treat context extension as a deployment-specific decision: - Measure the real input length distribution before enabling anything. - If long inputs are a minority, prefer chunking, retrieval or summarisation over stretching the whole deployment. - If long inputs are the workload, enable YaRN with the smallest factor that covers your p99 rather than the maximum the method allows. - If you have both workloads and they matter, run two deployments — one native, one extended — and route by input length. ## The memory bill Extending the window does not change the weights, but it changes what the serving engine must be able to hold. A 4× larger `--max-model-len` means a single sequence can occupy 4× the KV cache, so either startup fails for lack of blocks or your effective concurrency drops sharply. Long-context serving is a capacity decision as much as a quality one, and it usually arrives together with KV-cache quantisation or a smaller concurrency cap. ## What to say when asked The strong answer has three beats: name the mechanism and where it is configured; state that the extension is static and therefore has a short-prompt cost; and finish on the operational consequence — the extended context has to be paid for in KV cache, so you enable it for a deployment that needs it rather than as a default. A candidate who only recites the config block has learned a snippet; one who says "and I would not turn it on for a chat workload" has run one.