At 02:00 your policy decision service is down and every deploy is blocked - do you flip the default to allow?

## What is actually being asked A policy decision service is down. The gate is fail closed, so every deploy in the organisation is stuck, and it is the middle of the night. Somebody wants the flag flipped. The interviewer is not testing whether you know the flag exists; they are testing whether you can make a security decision under delivery pressure without either capitulating or grandstanding. ## Refuse the binary framing first "Do we open the gate?" is the wrong question because it assumes one dial. The decomposition is: - **Which rules?** Nearly all of them can degrade to allow for a few hours with acceptable risk. A small set cannot - the rules that stop something entering production with more privilege than it should have, where one miss is not recoverable by noticing later. - **Which pipelines?** Usually one team needs to move right now. Opening the default for everything to unblock one change multiplies the number of artifacts you will have to go back and assess. - **For how long?** An hour is a different decision from a week, and the answer must be encoded, not intended. - **Is there a better option?** If callers can evaluate a last-known-good rule set locally, most decisions are still real decisions and the question largely dissolves. ## The blast radius is the window, not the change The cost of flipping is not "one risky deploy". It is every change that ships while the default is flipped, none of which will have been evaluated, and - if you did not record them - which you will not be able to enumerate afterwards. That enumerability is the difference between a bounded incident and a permanent unknown. Before the flip goes in, decide where the list of affected deploys comes from. ## Make the reversion automatic The most common way this goes wrong is not the flip; it is that the flip outlives the outage. Somebody changes it at 02:00, the service comes back at 04:00, and the gate is still open in November. So: - the change carries an **expiry** and restores the previous default on its own; - it is **visible** - announced, alerted, on the dashboard - for as long as it is in force, because a temporary weakening that nobody can see is a permanent one; - it is **as easy to revert as it was to make**, so restoring is not itself a deploy through the gate you just opened. The circular case is worth calling out: if fixing the engine requires shipping a change through the gate the engine is blocking, you need a path that does not depend on the engine at all. Discovering that at 02:00 is a bad time to discover it. ## What you keep denying regardless The short fail-closed subset exists exactly so that this conversation has a floor. Whatever the pressure, rules protecting against privilege escalation on a workload keep denying, because the argument "we will look at it in the morning" does not undo a privileged workload that ran for eight hours. Having that floor written down before the incident is what lets an on-call engineer be firm at 02:00 without negotiating security architecture while sleep-deprived. ## The incident-fix case Often the blocked deploy *is* the fix for another incident. That sharpens the scoping rather than changing the principle: move that one change through a narrow path instead of opening the default for everyone, and have a human look at it, because the reason the machine is not looking at it is that the machine is down. ## Reconciliation When the service returns: 1. take the list of what shipped during the window; 2. evaluate those artifacts against the current rule set; 3. treat any violation as a normal finding with an owner and a date - not as a crisis, and not as something to quietly drop because it is already in production. Without step 1 the window stays unknown forever, and that gap is precisely the thing anyone reviewing the control will ask about. ## The postmortem writes the real answer The finding is not "the service went down". It is that an engine outage forced a security decision to be improvised, under pressure, by whoever happened to be on call. The actions are the design: the named fail-closed subset, the last-known-good fallback with an expiry, an out-of-band path to fix the engine, and a written statement of what degrades and what does not, agreed while everyone is calm. Treat the flip as the symptom. ## What a weak answer sounds like Two failure modes, and both are common. "Security first - we stay blocked until it is fixed" ignores that a frozen organisation is itself a risk and that the incident fix may be in the queue. "Just open it, we will tighten up tomorrow" ignores that tomorrow's tightening never gets scheduled and that nobody will know what shipped. The good answer scopes, time-boxes, records, and then fixes the design that made the choice necessary.