How would you ablate a 900-token prompt to find its load-bearing instructions?

## Why prompts accumulate dead weight A production prompt is rarely designed; it accretes. Someone adds a sentence after a bad demo, someone else pastes an edge-case rule after an incident, a third person adds two more few-shot examples, and six months later it is 900 tokens of instructions nobody can justify individually. That weight is not free: it costs tokens on every call, it competes for the model's attention, and — worse — it makes every future change unattributable, because you cannot reason about a change to a document whose parts you do not understand. Ablation is the procedure that turns that pile back into a set of components with known effects. ## The leave-one-out procedure 1. **Segment the prompt into named blocks.** Typical segmentation: persona/role framing, task statement, input description, decision rules or edge cases, output format specification, each few-shot example (or the example block as a unit), and any tone or safety guidance. Give each block an id so results are recordable. 2. **Fix everything else.** Same eval items, same model version, same decoding parameters, same scorer. The only variable across runs is which block is missing. 3. **For each block, delete it and rescore the identical items.** Record the paired delta — which items flipped from right to wrong, and which flipped the other way — rather than only the aggregate difference. Paired flips carry far more information than two accuracy numbers, and are far more sensitive on a small set. 4. **Rank the blocks by their measured contribution.** You now have an ordering from load-bearing to inert. A variant worth keeping in mind is **replacement instead of removal**: swap a block for a neutral placeholder of similar length. That separates "this content matters" from "the prompt got shorter", which occasionally matters for format and position effects. ## Interaction effects and the additive check Leave-one-out measures each block against the *full* prompt, so it systematically underestimates redundant blocks. If a rule appears both in the edge-case list and implicitly in a few-shot example, removing either alone changes nothing, and both look inert. Delete both and accuracy collapses. The cheap guard is an **additive rebuild**: start from the task statement alone and add back only the blocks that scored as load-bearing, then rescore. If the rebuilt prompt matches the original, the deletions were safe. If it falls short, some pair you dropped was jointly necessary, and you re-add candidates one at a time until you recover. The symmetric failure is a block that only matters *in combination* — a format instruction that is inert until the edge-case rule makes outputs longer. Additive rebuilding surfaces these too, which is why it is worth the extra run. ## Read the slices, not just the mean Aggregate accuracy hides the cases most rules exist for. A clause that says "if the message is a legal notice, never auto-respond" may touch four items in a 120-item set; removing it moves overall accuracy by three points at most and may move it by zero if those items were already handled. Score per failure mode or per segment, and treat any block that protects a high-cost slice as load-bearing regardless of its aggregate delta. If you cannot measure a block's effect because the eval set contains no instance of the case it guards, that is a gap in the eval set, not evidence that the block is useless — add items before you delete the rule. ## Cost, noise and when it is worth it Ablation costs one eval run per block, so a prompt with twelve blocks over a 120-item set is roughly 1,440 generations plus scoring — small for a text task, not free if each item involves a long context or a judge model. Two practical constraints follow. Ablate **blocks, not lines**, and only go finer on the blocks that turned out to matter. And check that your set can resolve the effect size you care about: if a single item is worth 1.7 points on a 60-item set, a two-point delta is not a finding. When runs are non-deterministic, repeat each configuration a few times and compare means, or the ablation just measures sampling noise. ## What to do with the result The output is not only a shorter prompt. It is a justified prompt: each surviving block has a measured contribution and, ideally, an eval case that fails when it is removed. That pairing is what makes future edits safe — the next engineer can delete a line and let the suite tell them whether it mattered, instead of leaving it in place forever because nobody knows what it was for.