The owner adds a preamble line telling a review bot never to discuss its tooling - what did that buy them?

## The situation A finding says a public review bot will describe its own operations to whoever asks in a pull-request comment. The owner adds a line to the preamble telling it not to discuss its tooling, re-runs the original request, sees a polite non-answer, and asks for the finding to be closed. The question is what that line actually changed. ## The part everybody says A line in a preamble is an instruction the model was trained to prefer, not a rule anything enforces. It holds most of the time and not all of the time, and the same phrasing behaves differently as the model or the surrounding context changes. Say it in one sentence and move on - it is assumed knowledge in this domain and it is not the interesting half of the answer. ## The part that decides the question Grant the strongest possible version of the fix: assume the line is honoured every single time. What is still true? **The surface did not move.** The bot still holds the same operations and still exercises them on every review. Nothing about what it can reach changed; one sentence about what it may say changed. **The bot's ordinary work product describes the surface anyway.** Consider what a public review reply contains: - *Attribution of what it read.* A review that refers to the failing job's output, the linked issue, and the diff has told the reader which sources this installation reaches. - *Refusal-by-explanation.* Asked to do something outside its wiring, an assistant naturally explains it cannot do that here - which is a statement about the surface, arrived at without discussing tooling in the abstract. - *Failure text.* When a step does not succeed, the reply commonly says something went wrong retrieving or posting something, which names the operation by its effect. - *Effects that are simply visible.* If it labels, links or comments, the reader watches it happen. None of that is the model discussing its tools. It is the model doing its job in public, and doing the job is itself an account of what the job includes. **The attacker's cost changed, and that is the honest finding.** Before, one comment produced a list. Now, the same map has to be assembled from behaviour across several interactions, at lower fidelity and with more inference. That is a genuine improvement in cost and should be reported as one. It is not an elimination. ## Reporting this without overclaiming or underclaiming Two failure modes, both common: | Failure mode | What it sounds like | Why it is wrong | |---|---|---| | Underclaiming | The finding is closed, the bot no longer answers | Tests one phrasing against one behaviour and reads a preference as a boundary | | Overclaiming | The fix is worthless | Ignores that the cheapest, most repeatable read really did go away | The accurate statement is narrow and defensible: *the direct disclosure route no longer returns the surface in one interaction; the surface remains inferable from the bot's normal public output, at higher cost and lower fidelity.* That sentence survives review because every clause is something you can point at. ## Verifying rather than assuming If you assert the residual, show it. The demonstration is not a cleverer request - it is reading what the bot already posts, unprompted, on ordinary pull requests: which sources it cites, how it phrases what it cannot do, what its failures say. That evidence is durable, it does not depend on a phrasing that may stop working next week, and it is what makes the difference between an opinion about preambles and a finding about this deployment. ## Where the residual genuinely shrinks Be fair about the other direction too. If the bot's replies stop attributing sources, stop explaining failures in the thread, and produce no visible effects, the behavioural route narrows sharply - because there is less work product to read. The residual exists in proportion to how much the bot says about what it did, which is a property of the deployment rather than of the preamble line. ## The interview signal An interviewer is checking whether a candidate can hold two things at once: that the change bought something real, and that it did not touch the mechanism the finding was about. Candidates who only debunk the preamble sound reflexive; candidates who accept the closure sound untested. The good answer prices the change and names the residual.