Trimming LLM chat history by character count overflows the window on Japanese threads and wastes 30% on English ones — how do you rebuild the budget?

## Diagnosing the symptom Two opposite failures from one policy is the signature of a unit mismatch. A character limit implies a fixed characters-per-token conversion, and that conversion is not fixed: English prose runs near four characters per token, while Japanese approaches one. Set the character limit so English fills the window and Japanese overflows it by a multiple; set it so Japanese fits and English leaves a third of the window — and a third of the available history — unused. There is no constant that satisfies both, which is why tuning the number is not the fix. A second, smaller contributor hides in the same policy. Even summing message strings perfectly, the count omits the chat template: per-message role and turn-delimiter tokens, the system prompt, and tool schemas re-sent every call. On a long thread of short turns that overhead alone can be a meaningful share of the request. ## The budget, stated properly Write it as an equation in tokens: `input_allowance = context_limit - reserved_output - safety_margin` `reserved_output` is the maximum response you will permit, and on models exposing a reasoning-depth dial it must also cover hidden reasoning tokens, which are generated and billed as output and consume the same window. `safety_margin` absorbs estimator error and anything the provider injects that you did not author. Anything left is what the rendered input may occupy. Then enforce it against the rendered request. Apply the model's chat template to the message list, tokenize the result with that model's tokenizer — offline where one is published, otherwise via a counting endpoint — and compare to the allowance. Now the Japanese thread and the English thread are measured in the same unit, and both use the window fully. ## Trimming, done carefully Trim at message boundaries, never mid-message and certainly never mid-token; a half-sentence at the head of the context is noise the model must reconcile. Keep the system prompt pinned. Drop oldest-first, and re-count after each removal rather than assuming the saving equals the message's own tokens, because removing a turn also removes its template overhead and any subsequent summarization changes the total again. Prefer summarizing dropped history over discarding it. A compact recap of the removed turns preserves commitments and decisions at a fraction of the tokens, and the summarization call itself needs a budget so it does not become an unbounded second cost. Handle the pathological case explicitly: if a single message exceeds the allowance on its own, chunk or summarize that message rather than letting the trim loop spin. One subtlety worth naming: trimming from the front changes the beginning of the prompt on every turn, which defeats reuse of a stable prefix downstream. Where that matters, prefer strategies that keep the earliest portion fixed — a pinned system block plus a summary slot that changes rarely — so churn is concentrated at the tail. ## Instrumentation, because this failed silently once already The original bug shipped because nothing measured it. Fix that alongside the algorithm. Log the pre-flight estimate next to the usage the response reports and alert when the residual drifts, which is what catches a new tool, an expanded system prompt or a model swap. Emit headroom as a metric segmented by locale, so a market approaching the limit is visible before it overflows. Track the trim rate — how often history is discarded — because a rise means users in some locale are losing context and quality even though no error is thrown. And put the worst-case locale into CI: a fixture thread of dense non-Latin text plus the largest toolset, asserted to fit within the allowance. ## The judgment being tested The interviewer wants three things. That you identify the unit mismatch rather than proposing a larger character limit. That you know the count must be taken over the templated request with the deployed model's tokenizer, not over concatenated strings with a remembered ratio. And that you treat the 30% waste as a real defect, not just the safe side of the bug — unused window is context the assistant could have used, so the English path is quietly delivering worse answers than it should. A strong answer also refuses a false economy. Counting tokens costs something: an offline tokenizer costs CPU, a counting endpoint costs a round trip. Both are cheap relative to a rejected request or a truncated conversation, and where the hot path cannot afford it, the fallback is a calibrated per-locale ratio at a pessimistic percentile plus a larger safety margin — an approximation you monitor, not a constant you trust.