Why should a policy test assert the returned message, not only that the verdict was deny?
answer
- deny is one bit, many causes
- right verdict, wrong branch
- the blocked engineer reads only this
- warn and deny are different outcomes
- assert the offending value, not the prose
basics
~20 sA verdict-only assertion passes when the rule denies for the wrong reason, so a broken condition stays hidden behind a green test. The message is also the whole product for the developer who is blocked, so it deserves an assertion of its own.
solid answer
~50 sDeny is one bit, and several different defects produce it. Suppose the case is an approved instance family in an unapproved region and my rule holds a typo in the approved-family list, so it denies on the family instead. The verdict matches, the test is green, and the rule now blocks every compliant change too. Asserting that the message names the offending region — and the allowed set — pins the reason, not just the outcome, and that is what turns the deny case into a real near miss. The message matters for its own sake as well: it is the only thing the blocked engineer sees at five o'clock, so "policy violation" is a failure of the rule even when the verdict is right. I assert the outcome level too, since a table that only distinguishes allowed from not-allowed cannot tell a warn from a deny.
go deeper
Know that a deny verdict alone does not say why the rule fired, and that a good denial message names the specific value that broke the rule.
Explain the failure concretely: a case expecting a region denial passes even when the rule actually denied on the family, so the near miss silently stops testing anything.
Show judgment on both fronts — pinning the branch and the outcome level in the assertion, while keeping the assertion loose enough on prose that copy edits do not train the team to ignore red.
Own the standard that a rule's message is a product surface with an owner: what every denial must contain, and how you hold rules to it across teams without turning the suite into a wording review.
## One bit is not enough evidence A validating decision reduces to a verdict, and a deny verdict is a single bit. Many different rules produce that bit for the same input: the rule you meant to write, a rule with an over-broad condition, a rule with a typo in a list so nothing matches, and a rule that denies unconditionally. If the assertion is `expect deny`, all four pass. Work the example. The policy is: only approved instance families, only approved regions. The fixture is an `m`-family instance in `ap-south-1`, and the expected result is a denial *because the region is not approved*. Now suppose the approved-family list was mistyped so that no family ever matches. The rule denies — for the family, not the region — and the test is green. The defect surfaces on the day someone submits an `m`-family instance in `eu-west-1`, which is entirely compliant and is blocked. The suite never had a chance to catch it, because it only ever checked the bit. Asserting on the message closes that: the case says the returned message must name `ap-south-1`. A denial produced by the family branch cannot satisfy it. The assertion has upgraded the case from "the rule said no" to "the rule said no for this reason", which is what a near-miss case is for. ## The message is the product There is a second, entirely separate reason, and it comes from a different chair. The person who meets your rule is not you; it is an engineer at the end of the day whose pipeline just went red. Everything they will ever know about the policy is in that message. A message reading "policy violation" or "denied by rule 47" turns a two-minute fix into a support ticket, and enough of those turn the gate into something teams route around. So a good deny message states four things: what was refused, which specific value offended, what the allowed set is, and how to proceed if the change is legitimate. For this rule: *"instance `web-3` requests region `ap-south-1`; approved regions are `eu-west-1` and `eu-central-1`. If this workload must run outside them, request an exception via <path>."* Once you accept that the message carries that much weight, testing it stops looking like pedantry — it is the user-facing surface of the rule and it regresses as easily as any other output. ## Pin the outcome, not just the verdict Many gates have more than two answers. A useful table for this rule looks like: | fixture | outcome | message must name | | --- | --- | --- | | approved family, `eu-west-1` | allow | — | | approved family, `ap-south-1` | deny | the region | | unapproved family, `eu-west-1` | deny | the family | | approved family, grandfathered `us-east-1` | warn | the region and the grandfathering | The fourth row is the one people forget. A legacy estate still runs in a region nobody wants new workloads in, so it returns a warning rather than a refusal. If the table only distinguishes "allowed" from "not allowed", that row is indistinguishable from the deny row, and a later refactor that collapses the warn branch into the deny branch passes the suite while silently blocking a team that was previously only nudged. Asserting the level — warn versus deny — is what makes the fixture pin the *behaviour* rather than the boolean. ## How far to go with message assertions The risk is brittleness: assert the message character for character and every copy edit breaks the suite, which trains people to update expectations without reading them. The workable middle is to assert on the load-bearing content — that the message contains the offending value and, where it matters, the allowed set or the identifier of the rule — and to leave the prose free to change. Substring or containment assertions over the specific value are enough to distinguish the region branch from the family branch, which is the whole point. A reasonable house rule: every deny case asserts the outcome level plus at least one value the message must contain, and that value is the one that makes this case different from its neighbours in the table. ## What an interviewer is listening for The weak answer treats message assertions as polish. The strong answer gives both reasons and keeps them separate — correctness (the verdict is ambiguous evidence, the message disambiguates which branch fired) and usability (the message is the only artefact the blocked engineer reads) — then adds the outcome level and shows awareness of the brittleness tradeoff.
- How do you assert a message without making the suite brittle to wording changes?Assert containment of the load-bearing values rather than the full string: the offending region, the resource identifier, and where it matters the allowed set. Those are the parts that distinguish one branch from another. Leave the surrounding prose unasserted so a copy edit does not turn the suite red and teach people to rubber-stamp expectation updates.
- Your table has a row expecting warn. What breaks if you only assert 'not allowed'?A change that collapses the warn branch into the deny branch keeps the suite green while turning a nudge into a block for whichever team relies on the grandfathered case. The row exists precisely to pin that distinction, so the assertion has to name the outcome level explicitly, not just the fact that the input was not clean.
- Should the message name the rule so it can be found later?Yes — an identifier for the rule and its owning team, alongside the offending value, is what lets a blocked engineer reach the right people and lets an operator correlate the denial with the rule that produced it. Assert the identifier as well, since it is the anchor everything else in the response gets found through.
saying these in an interview costs you the question
- Treats deny as sufficient evidence that the right branch fired
- Calls message wording cosmetic rather than tested behaviour
- Asserts the full message string character for character
- Cannot distinguish a warn outcome from a deny outcome in tests
- Writes messages that name no offending value