Why does a secret store partition that steady-state services barely notice break a service scaling out and a fleet on a rolling restart?

## Why steady state hides the failure Most consumers touch a secret store far less often than people assume. A service fetches the credentials it needs when it starts, holds them for as long as it runs, and then serves requests without going near the store again until something expires. That is exactly what makes the store survivable in normal operation — and exactly what makes a degraded store invisible. So during a partition at morning peak, the dashboards of a hundred running services are green. Their request rate against the store is close to zero. Nothing they do exercises the broken path. The store is a **start-up dependency**. Its failures are latent, and the latency is the gap between the failure starting and the next process starting. ## The two activities that convert a latent failure into an outage Both of these exist for good reasons, and both do the same dangerous thing: they create processes that have never fetched anything. 1. **Scaling out.** A customer-facing service adds instances because demand is climbing. Every new instance is a first fetch against a store that cannot serve it. The new instances fail to come up, the existing ones absorb the demand they were supposed to shed, and the scaling automation — seeing the load still rising — asks for more. The estate's response to peak becomes a stream of stillborn processes. 2. **A rolling restart.** A fleet of edge collectors is being replaced batch by batch. This is worse than scaling out in one specific way: it does not merely fail to add capacity, it **destroys capacity that was working**. Each batch takes processes that were happily holding values fetched before the partition and replaces them with processes that must fetch now. The failure spreads at the pace of the rollout. The read-only case sharpens both. Where a consumer starts by reading a value that is already stored, a read-only store serves it and the new process comes up fine. Where a consumer starts by asking for a credential the store generates per consumer, that is a write, and it fails instantly — so two services with identical restart behaviour have completely different outcomes, decided by which kind of credential they were designed to use. ## Why the automation keeps going The part that turns this into a long incident is that neither activity stops on its own: - The rollout marks an instance healthy on a signal that usually reflects the process itself, not whether it obtained what it needs from an external dependency. Where the new instance retries its start-up fetch for a while before giving up, the rollout can even count it as progressing. - The earlier batches genuinely succeeded, because they started before the partition. The operator watching sees a rollout that is going fine, right up until the batch that does not. - The scaling automation reads demand, not dependency health, so a degraded store makes it ask for *more* instances rather than fewer. ## What an operator actually does The store's state is usually not yours to fix in the first ten minutes, but the estate's rate of change is: - **Pause the rollout.** Every batch not yet started is an outage not yet caused. - **Hold the instance count.** Stop the scaling automation from manufacturing first fetches; the capacity you have running is the capacity you have. - **Do not restart the instances showing errors.** This is the reflex that turns a partial incident into a total one: a restarted process loses everything it was holding and joins the set that cannot come up. - **Find out which consumers need a generated credential to start**, because those are the ones that cannot come up even against a read-only store, and they will need the write path back before they can be moved at all. - **Know your expiry deadline**, since consumers holding something that will expire during the outage cannot renew and will restart into the same wall. What a restarting consumer *should* do when the store will not answer — refuse to start, or come up on a value it cached earlier, and for how long that copy stays usable — is a design decision that belongs to the consumer, and it is a separate subject from the store's own topology. What belongs here is the timing: the store's degradation and the estate's change rate multiply, and you control the second one. ## The point to make in an interview Say the store is a start-up dependency and that the estate's exposure is proportional to how many processes are starting, not to how much traffic it is serving. Then name the two activities that raise that number, say that a rolling restart also removes warm capacity, and give the first operator move: freeze the change rate before doing anything else.