skip to content

An operator widened a store rule by hand during an incident and the declared rule file was never updated — what happens at the next apply?

level: seniorimportance: should knowfreq 40%

answer

  1. declared text against live rules
  2. does the applier remove or only add
  3. it vanishes, or it ossifies
  4. when the apply runs decides when
  5. normalise both sides before comparing

basics

~20 s

It depends on the applier's design. One that converges on the declared set removes the hand edit silently, at an unrelated moment. One that only creates and updates what the file names keeps it forever, unreviewed.

solid answer

~40 s

There are two applier designs and they produce opposite outcomes. A **converging** applier makes live match declared, so the widening disappears at the next run — possibly weeks later, possibly breaking whatever came to depend on it, and the break will be blamed on the apply rather than the edit. An **additive** applier only creates and updates what the declared file names and never removes what it does not, so the widening lives on indefinitely and nothing ever reviews it again. Both designs are defensible; not knowing which one you have is not. The control that covers both is a drift report: a job that evaluates declared against live, normalises both, changes nothing, and tells a human what only exists on one side.

code

json · 14 lines
json
{
  "comparedAt": "2026-03-04T02:00:00Z",
  "onlyInLive": [
    {
      "grantedTo": "oncall-role",
      "names": "payments/*",
      "rights": ["read", "list"],
      "liveSince": "2026-02-19T01:14:00Z"
    }
  ],
  "onlyInDeclared": [],
  "differing": [],
  "changedNothing": true
}

go deeper

for a junior

Know that the rules live in the store and the declared file is a separate copy, so the two can disagree until something compares them.

for a middle

Explain the two applier designs and what each does with a live rule the file does not name, and why the comparison has to normalise both sides first.

for a senior

Reason about timing: on a proposal-triggered applier a hand edit can live for weeks and then vanish during an unrelated change, and the resulting break will be blamed on the wrong thing.

for a principal

Accept that people edit live during incidents and design the route that does not leave residue — a grant that expires on its own — rather than writing a rule telling them not to.

## Two appliers, two opposite outcomes "The declared file is the source of truth" describes an intention, not a mechanism. What actually happens to a rule that exists live but not in the file depends on how the applying job was written, and the two common designs disagree completely. | Applier design | The hand edit | The failure it creates | |---|---|---| | **Converging** — makes live match declared exactly | Removed at the next run | A grant vanishes at a moment unrelated to the incident; whatever came to rely on it fails then, and the apply gets the blame | | **Additive** — creates and updates what the file names, removes nothing | Survives indefinitely | A live grant nobody declared, nobody reviews, and nothing will ever surface | Neither is wrong. Converging gives you a real declaration and the ability to rebuild; additive is safer around rules the file does not manage. The defect is running one while believing you run the other. ## When the revert actually lands If you are converging, the next question is *when*. The answer is a property of the trigger, and the range is enormous: - an applier that runs on every proposed change may not run for weeks if nobody proposes one; - an applier on a short schedule runs within minutes, so the emergency widening barely outlives the incident; - an applier run only by hand runs when someone remembers. That range is the whole problem. On a proposal-triggered applier, the widening can be live and unreviewed for weeks, and then be removed by an unrelated change whose author has no idea they are touching it. The incident and its consequence are separated by so much time that nobody connects them. ## Why raw comparison reports drift that is not real The first attempt at a drift check compares the file's text to what the store returns and lights up permanently. Stores normalise what they store: they reorder, fill in defaults the file left out, expand or canonicalise a name pattern, and drop fields they do not use. A comparison has to run on the **meaning** of both sides — parse each into the same shape, apply the same defaults, sort, then compare. A drift report nobody believes is worse than none, because the real drift arrives inside the noise and is ignored with everything else. ## The drift report is the deliverable The job that closes this gap changes nothing. It evaluates both sides and produces three lists: 1. **Only in live** — a grant nobody declared. This is the emergency edit, and it is the finding that matters. 2. **Only in declared** — a rule that was written, approved and never reached the store, because an apply failed or was never run. 3. **Differing** — the same subject and names with different rights on each side. Then it makes someone decide. Each finding resolves one of two ways: **adopt** it into the declared file, where it becomes reviewable, or **remove** it deliberately, with someone knowing the moment it goes. Both outcomes are reviewed. Silent convergence and silent tolerance are the two ways to get an unreviewed outcome. ## The remedy is not "tell people not to edit live" People will edit live during an incident, because the alternative in the moment is staying locked out while something burns. The design answer is to give the incident a route that does not create drift — a grant that expires by itself, so the estate returns to the declared state without anyone remembering to undo anything. Designing that route is its own subject; what belongs here is the consequence: **a hand edit has no expiry**, which is precisely why it becomes drift and a self-expiring grant does not. ## Two related states worth naming - **The unapplied declaration.** The file was edited, approved and never applied. Everyone who reads the file believes a rule is in force that is not. This drifts in the opposite direction and the same report catches it. - **The second writer.** A job elsewhere that also writes rules — a provisioning step, a bootstrap script — will show up as permanent drift against a converging applier and produce a rule that flaps between two states. That is a fight between two owners, and it is resolved by ownership, not by comparison.

  • The applier converges, so the edit is eventually removed. Why is that still an incident?
    Because the removal lands at an arbitrary moment, triggered by an unrelated change, and anything that came to depend on the widening fails then. The break is attributed to the apply, not to the edit two weeks earlier. Meanwhile the grant was live and unreviewed for its entire lifetime.
  • What should the applier do when it finds live rules nobody declared?
    Report them and force a decision: adopt into the declared file, where the grant becomes reviewable, or remove deliberately, with an owner aware of the moment it goes. Silently converging and silently tolerating both produce an unreviewed outcome; the point of the report is that a human chooses.
  • Why does a naive text comparison of declared against live drown you in false drift?
    Stores normalise what they hold — reordering, filling in defaults the file omitted, canonicalising patterns, dropping unused fields. Comparison has to parse both sides into one shape, apply the same defaults and sort before diffing. A noisy report gets ignored, and the real finding is ignored with it.

saying these in an interview costs you the question

  • Assumes the next apply always reverts a hand edit
  • Assumes the applier leaves undeclared live rules alone
  • Treats an edited but unapplied file as a live change
  • Compares declared and live text directly and drowns in false drift
  • Thinks banning live edits during incidents is the fix