A rule tuned live in a purple-team session goes green - how do you prove it detects the technique, not the operator?

## The failure this question is about A rule tuned in a live session is fitted to a sample of one. The engineer has exactly one execution in front of them, the operator is waiting, and every condition added narrows the rule toward that single record. The result is a rule that goes green in the room and encodes the operator's tradecraft rather than the technique. ## Invariant versus incidental Take a cloud persistence step: mint a second access key for an existing identity, then attach an inline policy to it (ATT&CK `T1098.001`, additional cloud credentials). The audit records carry two very different kinds of field. **Incidental to this operator** - changes for free, next time: - the `userAgent` string, which names the client and its version - `sourceIPAddress` - the ordering of the two calls and the seconds between them - the target identity's name, the policy name, the exact policy document formatting - the region the console happened to be pointed at **Invariant to the technique** - hard to avoid while still achieving the objective: - the API actions themselves, and the fact that both occur against the same target identity - the caller creating credentials for an identity that is not the caller - the target already holding an active key, so this is an additional credential rather than a first one - the rarity of this principal performing credential creation at all, measured against the account's own history - the absence of the change from any change record or deployment window A rule leaning on the first list is an indicator rule wearing a behaviour rule's clothes. This is the indicator-versus-behaviour distinction with money on it: an artefact you observed is cheap for an adversary to change, a behaviour required by the objective is not. ## How you actually test it, in the room Ask for a variant that changes the field you most suspect the rule is leaning on. If you narrowed on the client, have the operator repeat the technique from the console or an SDK. If you narrowed on ordering, ask for the policy attach first, or a fifteen-minute gap between the calls. Best of all, have a *different person* execute it - somebody who was not watching the rule being written and will not unconsciously reproduce the shape it expects. A useful in-room habit: before the retest, the engineer states out loud which condition they expect the variant to break. If the rule survives a variant the engineer predicted would kill it, that is worth more than three greens on the original. ## The exit criterion The loop does not exit on the first green. It exits when a variant that nobody in the room designed the rule against fires - and if the second variant misses, the correct move is to **re-open the finding**, not to argue that the first result stands. The miss is the more informative result: the difference between the variant that fired and the one that did not is the specification for the next edit. This matters because of what happens to session output afterwards. A green from a session gets written into a report and read as coverage. If the only evidence behind it is one operator's execution, the organisation now believes it detects a technique it detects one performance of. ## Documenting honestly Write the variants down: which were executed, by whom, with which client, in which order, and which fired. State the ones you did *not* test. A finding that says *detected for CLI-driven executions by a single operator; console and delayed-ordering variants untested* is far more useful six months later than a green tick, and it tells the next person exactly which execution to try first. ## What interviewers listen for They want to hear you volunteer that a same-sitting green is weak evidence, name specific fields as incidental rather than gesturing at overfitting in the abstract, and describe a concrete second test. Candidates who describe the loop as *tune until it fires* have described the defect.