skip to content

In a MongoDB primary-secondary-arbiter replica set, what goes wrong when the secondary goes down?

level: seniorimportance: should knowfreq 42%

answer

  1. Three votes but only two data copies
  2. The arbiter stores nothing
  3. Majority acknowledgement needs data-bearing members
  4. Something stops advancing and memory grows
  5. The degradation shows up later, not immediately

basics

~20 s

The set keeps a primary, since the primary and arbiter are two of three votes, but the arbiter holds no data. So majority write concern can no longer be satisfied, the majority commit point stops advancing, cache pressure builds, and only one copy of the data remains.

solid answer

~50 s

A primary-secondary-arbiter set has three votes but only two data copies. Losing the secondary leaves the primary plus arbiter — still a majority, so the primary stays writable and `w: 1` traffic continues. Everything that depends on data-bearing acknowledgement stops, because an arbiter cannot acknowledge a write it does not store. Writes issued with `w: "majority"` block until `wtimeout` and fail. The majority commit point stops advancing, so the storage engine must retain history for majority-level reads and cache pressure grows steadily — a degradation that appears minutes to hours after the failure, not at the moment of it. And you are down to a single copy of the data, so the next failure is a data-loss event and the current failover exposure to rollback is at its worst. This is why current guidance is to prefer three data-bearing members over an arbiter.

code

javascript · 9 lines
javascript
// PSA: three votes, two data copies
rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "a:27017" },
    { _id: 1, host: "b:27017" },
    { _id: 2, host: "c:27017", arbiterOnly: true }
  ]
})

go deeper

for a junior

Know that an arbiter votes but stores no data, so a three-member set with one arbiter holds only two copies of everything.

for a middle

Explain why majority write concern becomes unsatisfiable when the single secondary is lost, and that election arithmetic and durability arithmetic count different things.

for a senior

Name the delayed failure mode — the frozen majority commit point and the resulting cache pressure — and treat 'secondary down' in a PSA set as a page, with explicit wtimeout so majority writes fail fast rather than hang.

for a principal

Own the topology decision: what the cost saving of an arbiter actually buys, what the degraded mode costs in an incident, and why three data-bearing members is the default recommendation.

## The configuration in question A primary-secondary-arbiter (PSA) set is a three-member replica set where one member is an arbiter: it participates in elections but stores no data. The attraction is cost — you get three votes, and therefore automatic failover, while paying for only two data-bearing servers. The arithmetic works. Three voters give a majority of two, so the set tolerates the loss of any one member and can still elect. The problem is that the arithmetic that governs *elections* is not the arithmetic that governs *durability*, and the arbiter is invisible to the second one. ## What actually happens when the secondary dies The primary and the arbiter are two of three votes, a majority, so the primary keeps its role. Nothing steps down. From a naive monitoring view the set looks fine: there is a primary, writes with default-ish concerns may keep succeeding, and the dashboard is green. Underneath, four things are now true. **Majority write concern is unsatisfiable.** A write acknowledged with `w: "majority"` must be applied by a majority of data-bearing voting members. There is now exactly one data-bearing member. The arbiter cannot acknowledge a write it never stores. Such writes block until their `wtimeout` elapses and then return an error — or block indefinitely if no `wtimeout` was set. Since MongoDB 5.0 the implicit default write concern is majority, though MongoDB adjusts that default in sets containing arbiters precisely because of this hazard; either way, any application code that explicitly asks for majority acknowledgement now fails. **The majority commit point stops advancing.** The majority commit point is the oplog position up to which a majority of data-bearing members are known to have replicated. With one data-bearing member left it freezes. The storage engine must keep enough history available to serve reads at the majority commit point, so it cannot discard old versions. Cache pressure grows, eviction gets harder, and performance degrades progressively. This is the failure mode that surprises people: it does not appear at the moment the secondary dies, it appears an hour later as mounting memory pressure and slowing queries. On earlier MongoDB versions the escape hatch was to disable majority read concern at startup; that option was removed in MongoDB 5.0, so on current versions the only real fix is to restore the second data-bearing member. **You are at one copy.** The set has no redundancy left. A disk failure or a corrupted file on the surviving primary is a restore-from-backup event, not a failover. **Rollback exposure is maximal.** With no secondary receiving the oplog, every write the primary accepts is beyond the majority commit point. If the primary then fails and the secondary returns and is elected, everything the primary accepted while alone is a rollback candidate. ## The mirror case: what if the primary dies instead? The secondary and the arbiter form a majority and elect the secondary as primary, so failover works — that is the whole reason the arbiter is there. But note what has happened: the newly promoted node was replicating asynchronously, so any writes the old primary had not sent are gone (rolled back when it rejoins), and you are again running on a single data copy. ## Why the arbiter is the wrong economy An arbiter provides exactly one thing: a vote. It provides no copy of the data, no read capacity, no ability to satisfy a data-bearing write concern, and no contribution to the majority commit point. In exchange it costs you a set whose degraded mode is far worse than a three-data-bearing set's degraded mode. Compare: in a set of three data-bearing members, losing one leaves two data copies, majority write concern still satisfiable (two of three is a majority), the commit point still advancing, and normal cache behaviour. The degraded state is genuinely a degraded state, not a slow-motion incident. That difference is why MongoDB's own guidance recommends against arbiters where a data-bearing member is possible, and why "we run PSA to save money" is a finding rather than a design in most production reviews. ## When a PSA set is nonetheless defensible Small, non-critical deployments where the data can be rebuilt, or environments where a third machine genuinely cannot be provisioned, can run PSA — provided the team knows the degraded behaviour, sets explicit `wtimeout` values so majority writes fail fast rather than hang, monitors the secondary's health as a first-class alert rather than a warning, and treats "secondary down" as a page rather than a ticket. ## What to monitor Alert on the secondary's state directly, not just on "is there a primary". Watch the lag of the majority commit point behind the primary's latest oplog entry, and watch cache utilisation on the primary. Any of those three going wrong in a PSA set means the clock is running.

  • Why does cache pressure build on the primary when the only secondary is down?
    The majority commit point cannot advance past what a majority of data-bearing members hold, and with one data-bearing member left it freezes. The storage engine must keep older versions available so reads at the majority commit point remain servable, so it cannot evict that history. Memory use climbs and eviction degrades. Restoring the second data-bearing member unfreezes the commit point and lets the retained history be released.
  • What is the recommended alternative to a primary-secondary-arbiter set?
    Three data-bearing members. The election arithmetic is identical — a majority of three is two — but losing one member still leaves two data copies, majority write concern still satisfiable, the majority commit point still advancing, and normal cache behaviour. The only thing you give up is the cost saving of running an arbiter instead of a real server.
  • If the primary in a PSA set fails instead of the secondary, does failover work?
    Yes. The secondary and the arbiter are two of three votes, so the secondary is elected. But the promotion has consequences: writes the old primary accepted and had not yet replicated are lost and will be rolled back when it rejoins, and the set is now running on a single data copy until the old primary returns.

saying these in an interview costs you the question

  • Says the arbiter can acknowledge a majority write
  • Assumes a green primary means the set is healthy
  • Thinks an arbiter adds a copy of the data
  • Blames the later cache growth on the workload, not the topology
  • Believes majority read concern can still be disabled at startup

context

resolving links…