How would you measure the usable context ceiling for your own workload?

## Why you have to measure it yourself Effective context is not a property of a model alone; it is a property of the model, the task shape and the material you surround the answer with. A single-fact lookup over clean prose survives far more fill than a twelve-fact reconciliation over near-duplicate documents. Published long-context results tell you what shape to expect — a band that holds, then a drop — but not where your knee sits. Since the budget you enforce in code is a specific number, and a wrong number produces either wasted headroom or silent quality loss, you measure. ## The sweep The design is a controlled experiment with exactly one independent variable: utilization. **Fix the task set.** Two hundred or so items is a reasonable target: enough that a few points of difference is not pure noise, small enough to run repeatedly at cost. Draw them from real production traffic where possible, and include the failure modes you actually care about, not only the happy path. Every item needs a programmatically checkable answer — an exact value, a set of clause identifiers, a schema-valid object — because a judge model adds its own variance to an experiment that already has plenty. **Fix everything except fill.** Same questions, same gold evidence present in every condition, same system prompt skeleton, same grader, same sampling settings. If the questions change between levels, the curve is meaningless. **Pad with realistic material.** Fill to the target utilization with documents from the same corpus — plausible neighbours that could compete with the correct answer. Random characters or lorem ipsum understate the difficulty badly, because discriminating against near-misses is a large part of what breaks at high fill. Padding with text from an unrelated domain is nearly as flattering: the model can dismiss it at a glance. **Choose levels that bracket the interesting region.** 20% as a low-fill reference, 50% and 80% through the band, plus one run close to the practical limit once output reservation is accounted for. If the drop lands between two levels, add a level between them rather than interpolating — the curve is cliff-shaped, so interpolation is exactly the wrong instinct. **Repeat each cell.** Generation is nondeterministic even at low temperature in most serving stacks. Three to five runs per level gives you a spread rather than a point, and lets you tell a genuine cliff from an unlucky run. Report the spread, not just the mean. ## Score more than accuracy An aggregate score hides the mechanism. Bucket the failures: - **Missed** — the model did not surface a fact that was present. - **Decoyed** — it surfaced a plausible but wrong passage, typically one of your padding documents. - **Reasoned wrongly** — it found the right evidence and drew the wrong conclusion. - **Truncated or refused** — an output-space or safety problem, not a retrieval problem. Each bucket points at a different remedy. Rising misses at high fill is the classic context-rot signature. Rising decoys says your padding is realistically confusable and your retrieval precision matters more than window size. Truncations mean your reservation arithmetic is wrong, not that the model degraded. ## Turning the curve into a budget Once you have the curve, the budget follows mechanically. Pick the utilization at which the metric you care about is still within your tolerance, subtract a margin for the fact that production inputs will be messier than your padding, and enforce that ceiling in the request builder: cap the number of retrieved chunks, cap history length, cap the size of tool results carried forward. Layer the output and reasoning reservation on top so the ceiling applies to input only. Then instrument production to log actual utilization per request, so you can see whether real traffic is drifting toward the region you decided to avoid. ## Maintenance The curve is not a permanent artifact. Re-run the sweep when: - You change model or model version. A larger advertised window is not permission to raise the cap; it is a reason to re-measure. - You change the prompt skeleton materially — a much larger system prompt or a big new set of tool definitions shifts where fixed cost ends and evidence begins. - You change retrieval depth, chunk size or ranking, since these change how much competing material lands in the window. - Your corpus changes character, for instance after ingesting a much larger or much more repetitive document set. Keep the harness cheap enough to run on that cadence. If a full sweep is expensive, run a reduced version — fewer items, three levels, three repeats — as a regression check, and reserve the full sweep for model migrations. ## What good looks like in an interview The answer that lands is a procedure, not an opinion: what is held fixed, what varies, how the padding is chosen, how many repeats, what is scored, and how the resulting number becomes an enforced cap with headroom. Saying "we measured it" without being able to describe the control is the tell that it was never measured.