In an Elastic Cloud deployment, how do availability zones and replicas provide high availability?
answer
- two knobs, two different places
- placement is not the same as redundancy
- check number_of_replicas before congratulating yourself
- even zone counts need a quorum helper
- size for surviving on n-1 zones
basics
~20 sZone count is infrastructure placement; replicas are what actually survive a zone loss. Elastic Cloud spreads instances across the zones you choose and keeps a master quorum possible, but an index with zero replicas still goes red when its zone fails.
solid answer
~50 sWhen you size a deployment you choose how many **availability zones** each tier spans — typically one, two or three. The platform places instances across those zones and configures shard allocation awareness so a primary and its replica never land in the same zone. With two zones it also adds a small **tiebreaker** node in a third zone so master elections still reach a quorum when one zone disappears. What the platform cannot do is create redundancy that your index settings do not ask for. `number_of_replicas: 0` on a three-zone deployment means a zone failure takes those shards with it and the cluster goes red — the zones only decide *where* copies live, not *whether* copies exist. HA therefore needs both: at least two zones **and** at least one replica, with enough spare capacity in the surviving zones to carry the load.
code
json · 9 linesPUT _component_template/ha-defaults
{
"template": {
"settings": {
"number_of_replicas": 1,
"index.routing.allocation.total_shards_per_node": 2
}
}
}go deeper
Recall that availability zones are chosen per tier when sizing the deployment, and that replicas are an index setting you control. Knowing both are needed is the key point.
Explain shard allocation awareness placing primary and replica in different zones, why an even zone count needs a tiebreaker for quorum, and what a red cluster means for searches and writes.
Show incident-shaped judgment: verify replica counts across templates, size so surviving zones absorb the traffic, expect recovery I/O to degrade performance after the infrastructure heals, and differentiate tiers.
Own the resilience-versus-cost policy: which datasets justify multi-zone plus replicas, which are reconstructible from upstream, what latency target must hold with a zone down, and how that is enforced through templates rather than left to whoever creates the index.
## The two independent knobs High availability on Elastic Cloud is the product of two settings that live in different places and are easy to conflate. The first is **deployment topology**: for each tier — hot, warm, cold, frozen, plus Kibana and machine learning — you choose an instance size and a **number of availability zones**. This is a plan-level, infrastructure decision made in the console or the deployment API. The second is **index-level redundancy**: `number_of_replicas`, set per index (usually through an index template or component template). This is a data-layer decision that lives in Elasticsearch itself and is entirely yours. Zones decide where copies are placed. Replicas decide whether copies exist. Neither alone gives you availability. ## What the platform does with the zone count When a tier spans multiple zones, the platform provisions instances in each of them and configures **shard allocation awareness** on the zone attribute. Allocation awareness makes Elasticsearch refuse to place a primary and its replica on nodes sharing the same awareness value, so with one replica and two zones, each shard has exactly one copy per zone. This is what converts a zone outage from data loss into a capacity loss. Master elections need their own arrangement. A cluster needs a majority of master-eligible nodes to elect a master; an even split across two zones would deadlock if either zone were lost. Elastic Cloud handles this by adding a small **tiebreaker** node in a third zone for two-zone deployments, so the surviving zone plus the tiebreaker still forms a majority. A three-zone deployment has a natural majority and needs no tiebreaker. A single-zone deployment is explicitly not highly available: a zone outage takes the whole thing offline regardless of replica settings, though replicas still protect against the loss of an individual node within that zone. ## The failure mode candidates miss The classic production incident is a deployment carefully sized across three zones whose indices were created with zero replicas — often to save disk on a large log dataset, or because an index template silently applied a legacy default. When a zone goes away, roughly a third of the primaries vanish, the cluster goes **red**, searches over those indices return partial results or fail, and indexing into affected shards stops. No amount of zone spreading helps, because there was never a second copy of those shards. The complementary mistake is running one replica and sizing capacity so tightly that the surviving zones cannot absorb the traffic. Losing a zone in a three-zone deployment removes a third of the compute; if the cluster was already at 80% CPU, the survivors saturate and the outage becomes a performance outage instead of a data outage. Real HA planning sizes for **n-1 zones**. ## Recovery behaviour after a zone returns When a zone comes back or the platform replaces the lost instances, Elasticsearch promotes surviving replicas to primaries and then rebuilds the missing copies by recovering from the remaining ones. That recovery consumes network and disk I/O and competes with live traffic, so a large cluster is degraded for a while after the infrastructure is nominally healthy again. Recovery throttling settings exist for exactly this trade-off, and knowing that recovery is not instantaneous separates operators from users. ## Choosing a number **Three zones with one replica** is the default posture for anything user-facing: survive a full zone loss, keep a quorum, keep every shard available. **Two zones with one replica plus the tiebreaker** is a cost compromise that still survives a zone loss but leaves less headroom. **One zone** is legitimate for development, for data reconstructible from an upstream source, or for a frozen tier whose data lives in object storage anyway. Note that more replicas also add search throughput, so the choice is not purely about resilience — but each replica multiplies storage cost, which on a metered service is immediately visible. ## Frozen and searchable-snapshot tiers A nuance worth mentioning: data in a frozen tier is backed by searchable snapshots in object storage, with nodes holding only a local cache. Losing such a node loses cache, not data, so redundancy requirements there are different from a hot tier holding the only copy of recent writes. Applying hot-tier replica thinking uniformly across tiers wastes money. ## Answering well State the split in one sentence — zones place copies, replicas create them — then give the failure scenario, the tiebreaker's role in two-zone deployments, and the capacity point about surviving a zone loss without saturating. That covers the ground an interviewer is probing.
- Why does a two-zone Elastic Cloud deployment get a tiebreaker node?Master election requires a majority of master-eligible nodes. Split evenly across two zones, losing either zone leaves exactly half — not a majority — and no master can be elected, so the cluster stops accepting changes. A small tiebreaker node in a third zone makes the surviving half plus the tiebreaker a majority, preserving elections without paying for a full third zone of data nodes.
- Your three-zone cluster runs at 75% CPU with one replica. What does a zone loss actually cost you?Data stays available because every shard has a copy elsewhere, but you lose a third of the compute while serving the same traffic, so the survivors go well past saturation and latency collapses. Meanwhile recovery of the missing replicas competes for I/O. Genuine HA sizing targets acceptable latency on n-1 zones, which usually means keeping steady-state utilisation well below two thirds.
- Does every tier need the same replica count?No. Hot-tier indices holding the only copy of recent writes need a replica. Frozen-tier data backed by searchable snapshots lives in object storage, so a lost node costs cache rather than data. Data that can be replayed from an upstream log or rebuilt from a system of record may justify fewer copies. Uniform replica policy across tiers usually overspends.
Zones are like storing your documents in three separate buildings. That only helps if you actually made copies; three buildings with a single original still means one fire destroys it.
saying these in an interview costs you the question
- Thinks choosing three zones alone makes the cluster HA
- Believes the platform adds replicas automatically
- Ignores that a lost zone also removes a third of capacity
- Assumes recovery after a zone returns is instant
- Applies hot-tier replica counts to frozen searchable-snapshot data