skip to content

Your platform team operates a secret store holding the payments team's database password — what rights does running the service actually require?

level: middleimportance: must knowfreq 58%

answer

  1. two objects: the service, the values
  2. operating acts on ciphertext
  3. backup, replica, upgrade need no plaintext
  4. restore is a write, not routine
  5. host access beats any rights split

basics

~20 s

Running the service and reading its contents are separate rights. Restart, replication, backup, upgrade and health checks all complete against the store's own ciphertext, so an operator identity gets those service-level rights and no read over any stored value.

solid answer

~50 s

A secret store is two things: a running service, and a set of values other teams own. Operating it means restarting instances, adding replicas, taking backups, upgrading the version and watching its health — every one of which acts on the process and on the store's own encrypted state, not on any value's plaintext. So the operator identity is granted those service-level rights and denied reads of value contents, while the payments team holds read over its own value and nothing over the service. Two edges matter. A `restore` is a write: it reinstates values as they were, including one a team has since replaced. And a backup puts the store's ciphertext in an operator's hands, so the split only holds while the key protecting that ciphertext is not theirs too — whose hands that key is in is its own question.

go deeper

for a junior

Recall that a secret store is both a service and a set of values, and that being allowed to run the service is not the same as being allowed to read what it holds.

for a middle

Explain the mechanics: which operations complete against ciphertext, why a restore counts as a write, and how a consumer authenticating with a value verifies it without anyone reading it.

for a senior

Show the judgment: enumerate the operations your team really performs, route restore and emergency reads onto their own recorded paths, and state honestly that host access defeats the rights split unless it is limited too.

for a principal

Own the tradeoff: how much operational latitude the platform team keeps, who holds the key protecting backups, and how the estate proves a year later that the split was in force rather than assumed.

## Two objects, two sets of rights A secret store is two things at the same time. It is a **running service**: a process to restart, replicas to add, state to back up, a version to upgrade, health to watch. It is also a **set of values that other teams own** — the payments team's database password among them. Separation of duties starts from the observation that those two things need two different sets of rights, and that most estates hand them out as one because a single administrative role was simply easier to grant. The platform team is measured on whether the store answers. The payments team is measured on whether its database password is current, correct and protected. Neither job requires the other's rights. ## What operating actually requires The way to settle the argument is to enumerate the operations and ask of each one whether it can complete without a value's plaintext. | Operation | Acts on | Needs a value's plaintext? | |---|---|---| | Restart or replace a failed instance | the process | No | | Add, remove or promote a replica | the store's own encrypted state | No | | Take a backup | the store's own ciphertext | No | | Upgrade to a new version | the binary and the state format | No | | Watch health, latency and error rates | the service's own signals | No | | Restore a backup onto a fresh instance | the store's ciphertext | No, but it is a write | | Confirm the payments password still opens the database | the value itself | Yes — and it is not the operator's job | Every row but the last completes against ciphertext, which is what makes this split practical rather than aspirational. Nobody is prevented from doing their job by it. - The operator identity holds lifecycle, replication, backup, upgrade and observation, and no read over any stored value. - The payments identity holds read over its own value, and nothing over the service. - Confirming that a value is **correct** belongs to the consumer that uses it: the payments service either authenticates against the database with it or it does not. That check is stronger than a human reading the value, and it needs no read right for anyone. ## Where the split leaks - **A restore is a write.** Putting yesterday's state back reinstates values as they were then, including one a team has since replaced — a withdrawn credential quietly returns and consumers start presenting it again. Restore deserves its own narrow grant and its own record, not membership in routine operation. - **A backup puts ciphertext in the operator's hands.** The split holds only while the key that protects that ciphertext is not also theirs. Who holds that key, and what stops one person from holding both halves, is a separate subject with its own answer. - **Rights are not physics.** Anyone with administrative access to the host running the store can reach the process's memory, where plaintext necessarily exists while the store is serving. A rights split must be paired with limits on host access, or it only constrains the people who were never going to bypass it. - **Debugging is how the read right comes back.** "Let me just look at the value to see what the service is getting" is the most common route by which an operator ends up holding read again — granted during an incident, never removed afterwards. ## What the split buys, and what it does not It bounds blast radius: a stolen operator identity yields a service someone can break, not an estate of credentials someone can sell. It makes the record answerable, because the identities that may read a given value are few and each has a reason to be there. When an external audit asks who could have read this value, the honest answer is a short list rather than a shrug. It does not stop denial — an operator can still stop the store, and everything downstream waits on it. It does not remove the need to replace values after an operator compromise, because host access and backup ciphertext are both in play. And it does not keep itself true: a grant that was right at the time drifts as tools arrive. ## Checking that the split holds 1. List every operation your team actually performed against the store last month, incidents included. 2. For each, decide whether it completes against the service and its ciphertext alone. 3. Grant the operator identity exactly those, and deny reads of value contents outright rather than by convention. 4. Give the residue — restore, and any genuine emergency read — its own narrow, recorded, time-boxed path instead of folding it back into the operator role. 5. Re-run the list whenever a new tool arrives, because tools tend to arrive holding whatever rights made them easy to install.

  • Why is restoring a backup treated as a privileged action rather than ordinary operation?
    Because a restore is a write. It puts values back as they stood at the backup's moment, which reinstates a credential a team has since replaced and withdrawn — consumers then read the old value and present it again. It also rolls back whoever owned what. So a restore gets its own grant, its own record and a check with the value's owner, rather than riding along with restart and upgrade.
  • The platform team says it needs read access to debug a team's failing service. What do you offer instead?
    Give them the signals, not the value: whether the store answered, which identity asked, at what time, and what result it returned. Almost every real debugging question is answered by that. Where the question genuinely is "is the stored value the one the database accepts", the answer comes from the consumer attempting authentication and reporting, not from a human reading it.
  • Does encryption at rest let an operator run the store safely without a rights split?
    No. At-rest encryption defends a stolen disk and a stolen backup file. It does nothing against a caller the rules allow, and nothing against someone with administrative access to the host, because the serving process necessarily holds plaintext in memory. The rights split and host-access limits are what keep an operator away from values; the ciphertext is not the barrier.

A removals firm can lift, transport, store and insure a locked filing cabinet without ever holding a key to it. Custody of the cabinet and ownership of the papers inside are two different jobs, and only one of them needs to read anything.

saying these in an interview costs you the question

  • Says the platform team needs read access to support the teams it hosts
  • Assumes encryption at rest keeps values away from whoever runs the service
  • Treats restoring an old backup as routine operation needing no review
  • Thinks holding a backup is harmless because it carries no plaintext
  • Claims an operator with no read right cannot cause an outage