skip to content

After a break-glass credential has been used during an outage, what must the post-use review produce?

level: seniorimportance: should knowfreq 34%

answer

  1. memory is not the record
  2. who, when, why, and what was reached
  3. read is not the same as replaced
  4. signed off by someone independent
  5. every use narrows the route or removes the need

basics

~20 s

A post-use review must produce a record of who opened the route, when and why, which values were actually read, whether each has since been replaced, and a sign-off by somebody who did not open it.

solid answer

~50 s

The review has two products, and a review that yields only the first is theatre. The first is a **record**: the identity that opened the route, the window's real start and end, the stated reason and its incident reference, and the list of values read — taken from the record kept by whatever served the reads, not from the opener's recollection. The second is a **change**. Every use should end in one of three findings: a value read and not yet replaced, so replace it; a route that reached more than the emergency needed, so narrow it; or an everyday gap that forced the open at all, so fix it and make the next incident not need the route. Somebody who did not open the route signs it off, and the tone is diagnosis rather than blame.

code

yaml · 20 lines
yaml
breakGlassUse:
  route: payments-database-owner-break-glass
  openedBy: individual-identity-of-the-engineer
  openedAt: "02:10"
  closedAt: "02:41"          # window ended on its own clock
  reason: "payments service down; needed the database owner credential"
  incidentRef: INC-4471
  valuesRead:                # from the holder's record, not from memory
    - name: payments/emergency/database-owner
      readAt: "02:16"
      replacedAt: "09:30"
    - name: payments/emergency/message-broker-admin
      readAt: "02:18"
      replacedAt: null       # <- this empty field is the finding
  reachableButNotRead:
    - payments/emergency/reporting-warehouse-owner
  reviewedBy: engineer-who-did-not-open-the-route
  findings:
    - "broker credential read and never replaced - replace before closing"
    - "route reaches the reporting warehouse for no emergency reason - narrow"

go deeper

for a junior

Remember that using an emergency credential is not the end: somebody writes down who used it, why, and what they saw, and somebody else reads that afterwards.

for a middle

Explain each field of the record and where it comes from, and why the list of values read must come from the serving system rather than the opener. Distinguish what was reached from what was read.

for a senior

Show the outcome side: a value replaced with a date and an owner, a route narrowed, or an everyday gap fixed so the next incident needs no route at all. Describe how you keep the review from becoming theatre.

for a principal

Set the standard: what every review must answer, who may sign one, how long the records are kept, and how you keep the tone diagnostic so engineers use the route instead of quietly keeping their own copy.

## What the review is for A break-glass route is the one path in the estate that deliberately hands a person more than their job normally allows, with no approver in front of them. The compensating control is entirely **after** the fact. That means the review is not paperwork attached to the control — the review *is* the control, and a route whose reviews are skipped whenever the reason was obviously good has no control at all. It produces two things: a record of what happened, and at least one change. ## The record | Field | Where it comes from | The question it answers | |---|---|---| | Opening identity | The authentication on the route, an individual not a shared account | Who is accountable for this window | | Opened at / closed at | The route itself | How long the rights were live, and whether the window closed on its clock | | Stated reason and incident reference | Entered at the moment of opening, mandatory | What this was for, in the words used at the time | | Values read | The record kept by whatever served the reads | What was actually taken, as opposed to what is remembered | | Values reachable but not read | The grant that defines the route | What was exposed to the window regardless of use | | Replacement status per value | The owner of each downstream system | Whether the copies taken still work | | Reviewer | A person who did not open the route | Whether anybody independent has actually looked | Two details carry most of the weight. First, the opener's account is a reconstruction written the morning after a night awake, and it will miss reads that turned out to be dead ends. The serving record will not. Second, the review has to distinguish **reached** from **read**: anything the window could have reached was exposed for the duration, so a route that reaches three credentials the outage never needed is a finding even when the log shows none of them were touched. ## The two findings that matter 1. **Narrow the route.** If the window reached credentials the emergency did not require, the route is wider than its purpose. Emergency routes drift wide for a comfortable reason — nobody wants to discover a missing right at 02:00 — and the review is the only place that pressure is ever pushed back. 2. **Remove the need.** The most valuable finding is that the route should not have been necessary: an everyday grant that was one right short, an automation that only a human could trigger, a runbook step that required an administrator for no structural reason. Fixing that converts a break-glass use into an ordinary operation next time, which is a strictly better outcome than making the route smoother. ## Reading is not the end of the story A window that closed on its time box ended the *rights*. It did nothing to the *values* that were read: those are now known to a human and, depending on how they were displayed, possibly to a terminal history, a screenshot, a chat message or a clipboard. Whether and how quickly each is replaced is a decision the review has to record explicitly, against the owner of the downstream system, with a date. A record with an empty replacement field is not a formatting problem — it is the finding. ## Who signs it, and in what tone The reviewer must be someone who did not open the route, because the point of the review is independence. The tone is load-bearing in a way people underestimate. If review is experienced as an investigation into the opener's judgement, the next engineer will avoid the route: they will find another way in, or keep a private copy of the credential so they never have to explain themselves. You would then have lost both the narrow grant and the visibility, which is the worst of the three available worlds. Review the route, not the person. ## What an interviewer is listening for That you name a record with a defensible source for each field, that you distinguish reached from read, that you treat replacement as separate from window closure, and that at least one finding changes something. A candidate who says "we file a ticket" and stops has described the theatre.

  • Why must the list of values read come from the holder's own record rather than from the person who opened the route?
    Because the opener is reconstructing, under pressure and usually hours later, and will omit the reads that went nowhere. The serving record is written by the system at the moment of each read, covers a second person who used the same open window, and is the only source that can be checked against something. The opener's account is still worth having - as the reason, not as the inventory.
  • What does a shared emergency account cost the review?
    Attribution. If the route authenticates a shared identity, the record answers "the break-glass account read this" and stops, so the review cannot say who acted, cannot ask them why, and cannot rule out a second person who used the same session. The fix is that individuals authenticate as themselves and are elevated into the route, so every line of the record names a person.
  • Which finding should narrow the route rather than replace a value?
    One where the window could reach credentials the emergency did not need. Everything reachable was exposed for the whole window regardless of what was actually read, so the correction is to the grant that defines the route, not to any individual value. Replacement answers a different question - what to do about the copies that were genuinely taken.

saying these in an interview costs you the question

  • The engineer's own account of what they read is sufficient
  • It was a genuine outage, so the review is a formality
  • The window closed, so the work is finished
  • Nothing needs replacing because the person who read it is trusted
  • The review exists to decide whether the opener was at fault