In a loop steered by a failing check, what should the agent be barred from editing, and what does barring it cost?
answer
- Evidence needs a still instrument
- Assertions, skips, thresholds, rule configuration
- Adding is the safe direction
- Locks turn machine turns into human ones
- Propose and report beats forbid
basics
~20 sWhatever decides the verdict should not move silently: assertions, expected values, skip lists, rule configuration, the threshold that fails the build. The cost is real — checks are sometimes wrong, and a locked loop burns turns failing against a stale expectation.
solid answer
~50 sA green result is evidence only insofar as the thing producing the verdict held still, so the parts that decide pass or fail are the candidates: assertions and expected values, the skip or exclusion list, the configuration saying which rules fail, and the threshold at which the build goes red. Implementation may move freely, and *adding* checks is safe in a way weakening them is not, since an added check does not make a failing case pass. The cost of a hard lock is that checks are sometimes wrong: a stale expectation the loop may not touch turns into turns spent failing against a wrong number, and a person is pulled in anyway. So the useful framing is rarely lock or unlock but **who authorises a move in the verdict** — the loop may propose one, provided it arrives separately and is reported rather than folded into the green.
go deeper
Know that a result is weaker evidence when the same run could change the thing that judged it, and that checks live in the working tree like everything else.
Explain which elements actually decide a verdict — assertions, skips, rule configuration, thresholds — and why adding a check is safe while weakening one is not.
Show the cost side: a locked check encoding a stale rule burns turns and pulls a person in anyway, so the practical setting is propose-and-report rather than forbid.
Own where the friction is worth paying — long unattended runs, silent failure modes, checks that are the only specification — and say plainly what your team treats a green run as evidence of.
## The property that makes a green result mean something An iteration loop produces two things: a change, and a verdict about it. The verdict is evidence only under a condition that is easy to leave unstated — **the thing producing it did not move while it was being satisfied.** A measuring instrument the subject can adjust measures the subject's willingness to adjust it. This is not a claim that agents behave badly. It is a claim about what a result can support. A green verdict reached over a fixed check set and a green verdict reached over a check set that changed during the run are different claims, and a team that cannot tell them apart has one kind of evidence where it thought it had another. ## What decides the verdict | element | verdict-deciding? | if it moves silently | |---|---|---| | assertions and expected values | yes | the verdict now reflects a different question | | the skip, exclusion or filter list | yes | the verdict is taken over fewer voters | | which rules a checker treats as failures | yes | complaints stop being raised at all | | retry or re-run settings for flaky cases | yes | a failure becomes a pass by repetition | | the threshold at which the build goes red | yes | the same results produce a different colour | | implementation, structure, naming | no | this is the work | | newly added checks | no | an addition does not make a failing case pass | The last row earns its place. **Adding** a check is the one direction that does not make a failing case pass — as long as the verdict means *nothing is failing* rather than *few things are* — which is why "the agent may add checks freely but may not weaken existing ones" is a workable asymmetry, and a much easier one to state than a full lock. ## What a lock costs Locking the verdict is not free, and a principal-level answer has to say so: - **Checks encode stale rules.** A price rule changes; the check does not. A loop that may not touch the expectation now fails forever against a wrong number, and every turn it spends doing so is wasted. - **It converts machine turns into human turns.** The intervention you avoided at the end arrives in the middle instead, and the person now has to reconstruct which of the two is wrong from a session they did not watch. - **It invites the loop to route around the lock.** If the assertion cannot move, the input can, or a branch can be added that makes the assertion trivially true. A lock on one element without a report on the rest moves the problem rather than removing it. - **It is friction on work where nobody cares.** A scratch tool, a one-file fix under a check you wrote an hour ago: the ceremony costs more than the risk. ## The move that usually beats a lock Between "the agent may change anything" and "the agent may change nothing that judges it" sits the setting most teams actually want: **the loop may propose a change to the verdict, but it must arrive as a separate, named thing rather than dissolved into a green result.** 1. **Require the run to say what it changed in the check set**, by name and by direction: this expectation moved, this case was skipped, this rule was switched off. 2. **Keep the pre-loop version of the failing case**, so "does the original case still pass" is a question anyone can answer afterwards. 3. **Treat a verdict change as a decision needing an owner** — settled by the rule that governs the behaviour, not by the run that wanted it. That preserves the evidence without pretending checks are infallible. What the team's review standard then requires of the change is a separate subject with its own answers. ## Where the friction is worth it There is no universal setting, and the judgement is what is being examined: - **Long unattended runs deserve more constraint than short supervised ones**, because nobody watched the middle and the report is all that survives. - **Blast radius beats novelty as the test.** Money, data destruction, anything whose wrongness is silent rather than loud: constrain. A failure that is immediate and visible carries its own alarm. - **Constrain harder where the check was the only specification**, since there is nothing else to compare against afterwards. - **Where the same loop runs many times a day**, a per-run report costs less than a policy nobody reads, and it accumulates into something you can actually look at. ## What this does not claim - **Not that a changed check is evidence of a bad run.** Corrected expectations are routine; the point is that the correction should be visible rather than implied by a colour. - **Not that a locked loop is a verified loop.** It removes one way a verdict can mislead you, not the gap between what the checks assert and what the behaviour should be. - **Not that every team needs this.** It is a cost, and on low-stakes work the cost is not repaid.
- Why single out adding checks as the safe direction?Because an addition leaves the failing case exactly where it was, still failing, while weakening, skipping and threshold changes all move the verdict instead. The exception is a verdict defined as a pass ratio, which is why the threshold counts as verdict-deciding too. Otherwise the asymmetry gives you a rule that is easy to state and to check.
- A team cannot enforce write scopes on its tooling. What is the cheap version?Require the run to report what it changed in the check set, by name and direction, and keep the pre-loop copy of the failing case so it can be re-run as written. Most of the value is in being able to tell the two kinds of green apart, which needs a record rather than enforcement.
- Is there work where none of this is worth the friction?Yes — short supervised loops on low-stakes code, where you watched the turns and the check was written minutes ago. The constraint earns its cost on long unattended runs, on anything whose wrongness would be silent, and where the check was the only written specification.
saying these in an interview costs you the question
- The agent should never be allowed to touch a test, in any circumstances
- Any expected value the agent edits is a covered-up failure
- A green run is a green run, whatever the run changed to get it
- Telling the agent the tests are off limits settles the question
- Locking the checks makes the loop's result trustworthy