During a network partition, why does a secret store's follower keep answering reads of a stored value while requests to generate a new credential fail?

middleshould knowfreq 42%

answer

one interface, two kinds of operation
reads answer from a local copy
issuance creates state, not a lookup
a write needs the write path
renewal writes a new expiry

basics

~20 s

Reading a stored value is served from the copy the follower already holds; generating a credential is a write — the store must record the new credential and its expiry before returning it, and a follower has no write path.

solid answer

~40 s

A follower holds a replicated copy of the store's encrypted contents, so answering a read means decrypting something that is already local — no other node has to be reachable. Generating a credential is not a lookup: the store creates or requests a credential in the downstream system and records what it issued, to whom, and when it expires, because nothing could later expire or withdraw it otherwise. That record is a write, and a node cut off from the write path cannot make one. The same is true of writing a value, renewing what was issued, and revoking it. So a partition usually produces a read-only store rather than a dead one: reads of what was already there succeed, and everything that changes state is rejected — immediately, not slowly.

A follower holds a replicated copy of the store's encrypted contents, so answering a read means decrypting something that is already local — no other node has to be reachable. Generating a credential is not a lookup: the store creates or requests a credential in the downstream system and records what it issued, to whom, and when it expires, because nothing could later expire or withdraw it otherwise. That record is a write, and a node cut off from the write path cannot make one. The same is true of writing a value, renewing what was issued, and revoking it. So a partition usually produces a `read-only` store rather than a dead one: reads of what was already there succeed, and everything that changes state is rejected — immediately, not slowly.

code

pseudocode · 10 lines

pseudocode

handle(request, node):
    if node.hasWritePath:
        return node.serveNormally(request)

    # partitioned from the writer, or a follower that never had the path
    if request.kind == READ_STORED_VALUE:
        return node.localCopy.get(request.name)   # may lag the writer

    # generate, write, renew, withdraw all end in a durable record
    return reject("no write path")

go deeper

for a junior

Know that a secret store is usually several nodes, that some of them only answer reads, and that asking the store to create something is a different kind of request from asking it for a value it already has.

for a middle

Explain the split mechanically: retrieval decrypts a local copy, while issuance, writes, renewal and withdrawal all end in a durable record and therefore need the node that accepts writes. Be able to sort a list of requests into the two piles.

for a senior

Show what the read-only mode does to a real estate: which consumers keep running, which cannot start, why the rejection is instant rather than slow, and why credentials that cannot be renewed give the outage a deadline of their own.

for a principal

Frame it as a contract. Decide which operations the estate is allowed to depend on during a partition, state that reads survive and state changes do not, and make consumers design against that promise rather than against the healthy case.

One interface, two kinds of operation

A secret store presents a single request surface, but what arrives on it divides cleanly in two, and the division is invisible until the store degrades.

Retrieval returns something the store already holds. The bytes are on disk in the store's own ciphertext, and answering means decrypting a local copy and handing it back. Nothing new is created.

State-changing requests end with the store holding something it did not hold before: a value that was written or overwritten, a credential it generated for a caller, a new expiry on something already issued, a withdrawal, a changed rule. Each of these has to be recorded somewhere durable before the store can honestly say it happened.

What a follower is, and what it cannot do

A store that an estate depends on rarely runs as a single process. One node (or one group of them) accepts writes; other nodes hold a replicated copy of the same encrypted contents and answer reads from it. That spreads read load and puts a copy near callers in another region — which is exactly why reads survive a partition that isolates those callers from the writer.

Designs differ in what a follower does with a state-changing request: some forward it to the node that owns writes and return that node's answer, so the caller never notices the topology until the writer itself is unreachable; others reject it and expect the caller to find the writer. No design lets a node that cannot reach the write path record new state on its own, so the outcome during a partition is the same either way.

Request	Needs the write path?	During a partition
Read a value written earlier	No	Answered from the local copy
Write or overwrite a value	Yes	Rejected
Generate a credential on request	Yes	Rejected
Renew something already issued	Yes	Rejected — the new expiry is state
Withdraw a credential	Yes	Rejected
Prove who the caller is	Depends on the design	Answered where verification records nothing; rejected where the store records a session it can later revoke

Why generating a credential is a write

Generating a credential on request is the operation most often misfiled as a read, because from the caller's side it looks like asking for a value and being given one. From the store's side it is the opposite. The store either creates the credential in the downstream system it is fronting or asks that system to, and then records locally at least three things:

who got it, so an access trail can answer who held what;
when it expires, so something can clean it up and so a renewal has a value to extend;
what has to be withdrawn later, so the credential can be taken away before its expiry.

Strip any one of those and the store has handed out access it cannot account for, cannot expire and cannot withdraw. That is why issuance is at minimum one local write, and usually a change in two systems.

The degraded mode this produces

The result is a store that is emphatically not down. It is read-only, and that shape has consequences worth naming:

A health probe that reads one value reports the store healthy for the entire outage, because the read path is the healthy one.
A process that starts by reading a stored value comes up normally; its neighbour that starts by asking for a freshly generated credential cannot start at all.
Rejection is fast and deterministic. A caller gets an immediate error, not a timeout, which is the clearest way to tell this state from a store that is merely slow.
Renewal is blocked, so credentials already issued keep marching toward their expiry with no way to extend them. The estate's real deadline is the shortest remaining validity anywhere in the fleet, which may be far shorter than the partition.
Withdrawal is blocked too, so a security response that depends on revoking a credential is held up by an availability failure.
A read may be behind the writer: a value rotated moments before the partition can still be served in its earlier form, and the caller has no way to tell from the response.

What to say when asked

Name the division first — retrieval against state change — then place issuance, renewal and withdrawal on the state-changing side and explain why. The candidate who says "replicas serve reads" has half of it; the one who can say which operations therefore disappear, and that the disappearance is instant rather than slow, has the part that matters when the estate is on fire.

## One interface, two kinds of operation A secret store presents a single request surface, but what arrives on it divides cleanly in two, and the division is invisible until the store degrades. **Retrieval** returns something the store already holds. The bytes are on disk in the store's own ciphertext, and answering means decrypting a local copy and handing it back. Nothing new is created. **State-changing requests** end with the store holding something it did not hold before: a value that was written or overwritten, a credential it generated for a caller, a new expiry on something already issued, a withdrawal, a changed rule. Each of these has to be recorded somewhere durable before the store can honestly say it happened. ## What a follower is, and what it cannot do A store that an estate depends on rarely runs as a single process. One node (or one group of them) accepts writes; other nodes hold a replicated copy of the same encrypted contents and answer reads from it. That spreads read load and puts a copy near callers in another region — which is exactly why reads survive a partition that isolates those callers from the writer. Designs differ in what a follower does with a state-changing request: some forward it to the node that owns writes and return that node's answer, so the caller never notices the topology until the writer itself is unreachable; others reject it and expect the caller to find the writer. No design lets a node that cannot reach the write path record new state on its own, so the outcome during a partition is the same either way. | Request | Needs the write path? | During a partition | |---|---|---| | Read a value written earlier | No | Answered from the local copy | | Write or overwrite a value | Yes | Rejected | | Generate a credential on request | Yes | Rejected | | Renew something already issued | Yes | Rejected — the new expiry is state | | Withdraw a credential | Yes | Rejected | | Prove who the caller is | Depends on the design | Answered where verification records nothing; rejected where the store records a session it can later revoke | ## Why generating a credential is a write Generating a credential on request is the operation most often misfiled as a read, because from the caller's side it looks like asking for a value and being given one. From the store's side it is the opposite. The store either creates the credential in the downstream system it is fronting or asks that system to, and then records locally at least three things: - **who got it**, so an access trail can answer who held what; - **when it expires**, so something can clean it up and so a renewal has a value to extend; - **what has to be withdrawn later**, so the credential can be taken away before its expiry. Strip any one of those and the store has handed out access it cannot account for, cannot expire and cannot withdraw. That is why issuance is at minimum one local write, and usually a change in two systems. ## The degraded mode this produces The result is a store that is emphatically not down. It is **read-only**, and that shape has consequences worth naming: - A health probe that reads one value reports the store healthy for the entire outage, because the read path is the healthy one. - A process that starts by reading a stored value comes up normally; its neighbour that starts by asking for a freshly generated credential cannot start at all. - Rejection is **fast and deterministic**. A caller gets an immediate error, not a timeout, which is the clearest way to tell this state from a store that is merely slow. - Renewal is blocked, so credentials already issued keep marching toward their expiry with no way to extend them. The estate's real deadline is the shortest remaining validity anywhere in the fleet, which may be far shorter than the partition. - Withdrawal is blocked too, so a security response that depends on revoking a credential is held up by an availability failure. - A read may be behind the writer: a value rotated moments before the partition can still be served in its earlier form, and the caller has no way to tell from the response. ## What to say when asked Name the division first — retrieval against state change — then place issuance, renewal and withdrawal on the state-changing side and explain why. The candidate who says "replicas serve reads" has half of it; the one who can say which operations therefore disappear, and that the disappearance is instant rather than slow, has the part that matters when the estate is on fire.

The store proves who every caller is before it serves anything. Can a follower do that during the partition?
It depends on the design, and this is the case people get wrong. Where the store verifies the caller against material that is already replicated and records nothing, verification works on a follower. Where the store records a session it can later withdraw, that record is a write — so proving who you are fails, and the store looks completely dead to a new caller even though existing callers' reads are being answered.
A read on a follower returns a value the writer replaced two minutes ago. Is the store broken?
No — it is serving a copy that has not caught up, which is a property of replication rather than an error. The problem is that the caller cannot tell: it gets a valid-looking value with no marker that it is behind. That is why a rotation is not finished when the writer accepts it, and why consumers that must see the newest value have to be routed to the node that owns writes.

saying these in an interview costs you the question

Says a replica can serve everything because all the data is there
Treats generating a credential as a lookup of something already stored
Assumes a passing read probe proves the store is fully serving
Calls renewal a read because no new credential is minted
Thinks a follower rejects writes because it lacks the protecting key

During a network partition, why does a secret store's follower keep answering reads of a stored value while requests to generate a new credential fail?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

One interface, two kinds of operation

What a follower is, and what it cannot do

Why generating a credential is a write

The degraded mode this produces

What to say when asked

▾common follow-ups2 questions

▾red flags5 flags

One interface, two kinds of operation

What a follower is, and what it cannot do

Why generating a credential is a write

The degraded mode this produces

What to say when asked

answer

go deeper