Every service now starts against one secret store — how do you decide how much of the estate's availability to stake on it?
answer
- two levers, not one
- followers buy reads, not writes
- every copy widens the footprint
- name what must survive a partition
- the store inherits its own dependencies
basics
~20 sTwo levers exist: make the store more available, or make the estate need it less. Decide which operations must survive a partition — plain reads — and which may fail, then publish that as the contract every consumer designs against.
solid answer
~50 sStart by separating the operations. Adding followers buys read availability and nothing else: the write path stays exactly as available as it was, so issuance, renewal and withdrawal still stop. Spreading the write path across regions trades write availability during a partition for surviving the loss of a region, and every extra copy widens the footprint of the material you are protecting. The second lever is usually cheaper: reduce what the estate demands. Fewer consumers that need a generated credential merely to start, longer validity where the exposure is acceptable, and start-up paths that read rather than write. Then state the promise plainly — "reads of an already-stored value survive a partition, anything that changes state does not" — and make teams design to it. The honest ceiling is that the store is never more available than whatever it needs to unlock itself.
go deeper
Know that a store everything starts against becomes a single dependency for the whole estate, and that adding copies of it is a design decision with costs rather than an obvious improvement.
Explain why replicas help reads and not writes, and why the operations that change state — issuance, renewal, withdrawal — are the ones that disappear first when the store degrades.
Show how you would reduce what the estate demands: which consumers need a generated credential to start, which validity windows make renewal a per-hour dependency, and how deployment policy changes the exposure.
Make the trade explicit and write it down. Say which operations must survive a partition, what security cost you accepted to get there, who agreed, and how you would show an external reviewer that the failure modes were chosen rather than discovered.
## The question behind the question Once every service reads a credential before it can serve a request, the store's availability is the estate's availability, and "make it highly available" is not an answer — it is a budget request with no number in it. The decision a lead actually owns is *which operations the estate is allowed to depend on during a failure*, and what it will pay for each one. ## Lever one: make the store more available This lever is real but far more limited than it sounds, because availability is not one quantity here. - **Followers in more places buy reads, not writes.** More copies close to more callers means more reads survive more failures. The node that accepts writes is exactly as available as it was, so issuance, renewal and withdrawal are untouched by the investment. - **Spreading the write path across regions is a genuine trade, not a free win.** A write path that must reach agreement across distant nodes survives losing one of them, and stops during a partition that separates them. You are choosing which failure you would rather have, not removing failure. - **Every copy widens the footprint.** A replica is another set of nodes holding the estate's ciphertext, another set of operators, another place for a backup to leak from. Availability engineering here has a security cost that pure capacity work does not. - **The store inherits its own dependencies.** If a node needs a key it fetches from somewhere to open its contents at start-up, that somewhere is now in your start-up path too, and the estate's floor is the lower of the two. A design that requires people to supply the protecting key adds a human to the floor. ## Lever two: make the estate need it less Usually the cheaper lever, and the one teams skip. 1. **Move start-up paths off the write path.** A consumer that needs the store to *generate* something before it can serve cannot start against a read-only store. A consumer that reads an already-stored value can. Knowing how many of each you have is the single most useful inventory here, and moving a consumer from the first group to the second buys availability without touching the store at all. 2. **Choose validity windows deliberately.** Longer-lived credentials mean fewer renewals, so fewer operations that fail when the write path is gone — paid for with a longer window in which an exposed value still works. That is a security-against-availability trade, and it is a lead's to make explicitly rather than a default to drift into. 3. **Bound how many processes start at once.** Exposure is proportional to the number of start-ups happening, so deployment and scaling policy is availability policy. Deciding whether rollouts may proceed while the store is degraded is part of this answer. ## What a published contract looks like The deliverable is a statement other teams can design against, in the store's own vocabulary rather than an uptime figure: | Operation | Promise during a partition | What a consumer must therefore do | |---|---|---| | Read a value already stored | Expected to keep working | May depend on it at start-up | | Write or overwrite a value | Expected to fail | Must not be on a serving path | | Generate a credential on request | Expected to fail | Start-up must not require it, or the service accepts the coupling knowingly | | Renew what was issued | Expected to fail | Validity must exceed a plausible outage | | Withdraw a credential | Expected to fail | Emergency response has to plan for the delay | The last row is the one that makes this a security decision and not just an availability one: if withdrawal requires the write path, then a topology chosen for availability also decides how quickly you can take access away. A team that buys write availability at the cost of nothing else has usually not noticed it was buying revocation speed too. ## The judgment, stated honestly There is no configuration that makes a shared dependency free. The defensible position is a stated one: name the operations that must survive, invest only in those, tell every consumer team what to expect, and be able to show an external reviewer which failures were accepted deliberately and by whom. The weak answer promises the store will not be read-only. The strong one says read-only is a state the store will certainly enter, describes exactly what the estate can still do while it lasts, and names who agreed to that. ## What this is not This is not a question about whether an individual service caches its credentials and for how long — that is the consumer's own design. It is about what the platform promises and what it asks the estate to give up in return.
- What is the availability ceiling you cannot buy your way past here?Whatever the store itself needs in order to serve. If a node obtains the key that protects its contents from another service at start-up, that service is in the estate's start-up path and the floor is the lower of the two. Where people must supply that key instead, the floor includes how quickly someone can be reached. Counting the store's uptime without counting what it depends on overstates the number.
- Why is lengthening credential validity a real availability lever, and what does it cost?Renewal is a write, so consumers holding something long-lived survive a partition that stops the write path, while consumers renewing hourly fall over within the hour. The cost is exposure: a leaked credential stays useful for exactly as long as you extended it, and withdrawal may itself be unavailable during the same outage. It is a deliberate trade, not a tuning value.
- A team proposes a separate store per region to remove shared fate. What do you weigh?Shared fate falls, and everything else rises: more copies of the estate's material, more backups to protect, more operators with access, and a harder job answering who could read a given value. You also have to decide what happens to a credential issued in one region and used in another. It can be right, but it is a blast-radius and operations decision as much as an availability one.
saying these in an interview costs you the question
- Answers 'make it highly available' with no cost named
- Assumes more replicas also raise write availability
- Counts the store's uptime without counting what unlocks it
- Treats longer credential validity as free availability
- Promises the store will never go read-only
- Ignores that a topology choice also sets withdrawal speed