skip to content

A policy gate denied every entry of a CI matrix job. How do you tell a rule defect from a real violation?

level: seniorimportance: should knowfreq 54%

answer

  1. everything denied at once is a clue
  2. diff evaluated documents, not sources
  3. present and wrong, moved, or absent
  4. absence has two meanings
  5. perturb the field, then add a fixture

basics

~20 s

Diff the documents the engine actually evaluated against the shape the rule author assumed. If expansion produced entries where the field the rule reads is absent rather than wrong, the rule met an input it was never written for.

solid answer

~50 s

I pull the evaluated document for a denied entry and for one that passed elsewhere, and diff them. A matrix expands one authored block into several concrete jobs, and expansion can drop, rename or re-nest fields, so the rule may be reading a path that does not exist on these entries. That distinction is the question: a real violation declares something the rule forbids, while a rule defect is a job whose shape the rule was never written for. Take a rule requiring any build job that can reach production credentials to declare an egress allowlist. If the entry genuinely holds credentials and declares no allowlist, the denial is correct. If expansion attached a shared credential block, or moved the allowlist under a path the rule does not read, the rule is at fault. I confirm by inserting the field at the path the rule reads and replaying.

code

json · 13 lines
json
{
  "entry_smoke": {
    "job": "integration-tests",
    "credentials": ["prod-deploy-token"],
    "network": { "egress_allowlist": ["registry.internal", "artifacts.internal"] }
  },
  "entry_contract": {
    "job": "integration-tests",
    "matrix": { "suite": "contract", "region": "eu" },
    "credentials": ["prod-deploy-token"],
    "network": {}
  }
}

go deeper

for a junior

Know that a rule firing does not by itself mean the change was unsafe, and that the document the engine judged can differ from the file a person wrote.

for a middle

Be able to walk the diff between an evaluated denied document and a passing one, and to name the three findings it can produce: forbidden value present, field relocated, field absent.

for a senior

Show judgment about which side to fix, prove the hypothesis by perturbing the replayed input, and close the loop by adding the surprising document to the rule's fixtures.

for a principal

Own the principle that a rule which cannot distinguish 'declared nothing' from 'this surface does not express that' will keep producing false denials, and that repeated false denials cost more trust than the rule buys.

## Two very different problems wearing the same message A denial tells you a rule fired. It does not tell you whether the world is wrong or the rule is wrong. The two outcomes need opposite responses - one changes the job, the other changes the rule - and the cost of guessing is high in both directions. Wave through a real violation and the guardrail was theatre; force a team to contort a correct job around a rule defect and you have taught them that policy is an obstacle to route around. When an entire matrix job is denied at once, the prior shifts sharply toward rule defect. A single denied entry among many usually reflects something specific to that entry. Everything denied at once means the rule is reacting to a property of the *shape* the expansion produced, not to a property any human authored. ## Reconstruct what the engine read The authored block is not what was judged. A matrix takes one block and produces N concrete jobs, and along the way it interpolates variables into names, attaches shared blocks such as credential or service definitions, applies platform defaults, and sometimes flattens or re-nests structures. So the artifact to look at is the evaluated document for a denied entry. Then get a contrast case: the same rule passing somewhere else, ideally a non-matrix job of the same kind. Diff the two documents. You are looking for one of three findings: - **The field is present with a forbidden value.** Real violation. The job declares something the rule forbids. - **The field is present but at a different path.** Rule defect. The author wrote against the shape they saw in the source, and expansion moved it. - **The field is simply absent.** This is the interesting one, and it is where most of these investigations land. ## Absent is not the same as wrong Consider a rule that says any build job able to reach production credentials must declare an egress allowlist. The author pictured a job with a `credentials` list and a `network.egress_allowlist` list, and wrote the rule to deny when credentials are present and the allowlist is missing or empty. Now the matrix expands. One entry inherits the shared credential block - so the rule's trigger condition is satisfied - while the allowlist, which the author declared once at the top of the source block, did not survive expansion into every entry, or landed under a per-entry `network` object that the expansion created empty. Every entry now looks, to the rule, like a job holding production credentials with no allowlist. The rule fires correctly on the document it was given and refuses work that is not actually dangerous. The diagnostic point is that *absent* carries two contradictory meanings which the rule cannot distinguish: "this job deliberately declares nothing" and "this document does not represent that concept at all". A rule that treats absence as a violation will be right about the first and wrong about the second, and it will be wrong every time the surface it reads changes shape. This is why the reconstruction has to start from the evaluated document rather than from an argument about intent. ## Confirm the hypothesis before you act Do not stop at a plausible story. Replay the denied document off the enforcement path, then perturb it: insert the allowlist at exactly the path the rule reads and confirm the denial clears. If it clears, you have proven both what the rule wants and where it expects it. If it does not, your hypothesis is wrong and there is a second condition you have not found. The same perturbation tells you which fix is right. If the only way to clear the denial is to write a field that has no meaning in the expanded form, the rule is asking for something the surface cannot express, and the rule must change. If the field is meaningful and the team simply did not set it per entry, the job changes. ## What the fix looks like on the rule side When it is a rule defect, the repair is usually not "exclude matrix jobs". It is to make the rule's trigger and its check read the same subject: decide deliberately what the rule does when the field is absent, and either look for the concept everywhere it can legitimately live in this surface, or narrow the trigger so the rule only fires on documents whose shape it actually understands. Then add the expanded document that caused the denial to the rule's fixture set, so the shape the author never pictured is now a shape the rule is tested against forever. ## And say so out loud Whichever way it lands, write the finding in one sentence - *this rule denied because the expanded entry has no field at path X, which the source declares once at the block level* - and put it where the blocked team can read it. A team that watched a gate refuse six jobs will remember whether the answer arrived as an explanation or as a shrug.

  • Why does the whole matrix being denied point toward a rule defect rather than a violation?
    Because a genuine violation is usually a property of one authored decision, and expansion multiplies one block into many entries that differ only in the matrix values. When every entry fails identically, the rule is reacting to something the expansion produced for all of them - a missing or relocated field - rather than to something a human chose. It is a prior, not a proof, so you still diff the documents.
  • What do you change on the rule once you have confirmed it was reading a path that does not exist after expansion?
    Make the trigger and the check read the same subject, and decide explicitly what absence means for this rule rather than letting it fall out of the syntax. Then commit the expanded document that caused the denial as a test fixture, so the shape the author never pictured is permanently covered. Excluding matrix jobs wholesale is the wrong repair - it removes the control from exactly the jobs that hold credentials.
  • The blocked team wants to disable the rule for their pipeline until it is fixed. How do you respond?
    Separate urgency from scope. If the rule is genuinely misfiring, the honest short-term move is to make it non-blocking for the shape it does not understand, not to switch it off for that team's pipeline, since the same pipeline may also contain jobs the rule judges correctly. Fix within hours, keep the finding visible, and let the narrowed rule expire when the repair lands.

saying these in an interview costs you the question

  • Assumes the gate is right because the rule ran successfully
  • Reads the authored source instead of the expanded documents
  • Treats a missing field as proof of a violation
  • Adds an exclusion for matrix jobs instead of fixing the rule
  • Fixes the rule without adding the surprising document as a fixture

context