skip to content

Cluster State, Allocation & Recovery

Node roles, master election, and shard allocation decide whether the cluster is green, yellow, or red, and recovery is what copies shards back after a node leaves. Interviewers ask you to debug a red cluster, which really means naming the allocation explain API.

part ofElasticsearchoverview, primer and where to startread it →
on this pageshow

questions

6

What do green, yellow, and red cluster health mean in an Elasticsearch cluster?

level: juniorimportance: must knowfreq 80%

answer

  1. It is about shard copies, nothing else
  2. Two of the three colours still serve all data
  3. Ask which copy is missing: primary or replica
  4. Cluster colour equals the worst index colour

basics

~20 s

Green: every primary and replica shard is assigned. Yellow: all primaries are assigned but at least one replica is not. Red: at least one primary is unassigned, so part of the data is missing from search results and writes to that shard fail.

solid answer

~40 s

Cluster health is purely a **shard-assignment** report. **Green** means every primary and every replica shard has a node to live on. **Yellow** means all primaries are assigned, so all data is searchable and writable, but at least one replica is unassigned — you have full data, reduced redundancy. **Red** means at least one primary is unassigned: that shard's data cannot be searched and indexing into it fails, while the rest of the index still works. Each index gets a colour and the cluster colour is the worst index colour, so one broken index makes the whole cluster red. A single-node development cluster with default settings is permanently yellow, because a replica is never allowed onto the same node as its primary. `GET _cluster/health?level=indices` shows which index is at fault.

code

bash · 1 line
bash
GET _cluster/health?level=indices

go deeper

for a junior

Memorise the three definitions in terms of primary and replica shards, and be ready to explain why your laptop cluster is always yellow.

for a middle

Explain why a replica is never placed on its primary's node, and show how to go from a colour to the individual unassigned shard using the health and _cat APIs.

for a senior

Show the diagnosis path from a red cluster to a root cause, and discuss alerting: which colours page, which merely warn, and why duration matters more than the colour itself.

for a principal

Frame health colour as one signal in an availability SLO. Argue about what redundancy level the business is actually paying for and how many simultaneous node losses each index tier should survive.

## What the colours actually measure Elasticsearch health colour answers exactly one question: *are all the shard copies this cluster is supposed to have actually allocated to a node?* It says nothing directly about CPU, latency, disk, or JVM pressure. Every index is built from a fixed number of primary shards, and each primary can have replica copies. The cluster's job is to find a node for every one of those copies. The colour reports how far it got. ## Green Green means every primary and every replica of every index is assigned to a node. Full data availability, full redundancy: you can lose a node and still have a copy of everything. Green is not a statement about performance — a green cluster can be at 92% disk usage and rejecting requests from queue saturation. ## Yellow Yellow means all primaries are assigned but at least one replica is not. Nothing is missing: every document can be searched and every write succeeds. What you have lost is redundancy — if the node holding an unreplicated primary dies now, that shard goes unavailable and the cluster turns red. Yellow is therefore a "fix it soon", not a "wake up at 3am" (unless it lingers). The most common yellow is not a fault at all. A one-node cluster that creates an index with one replica stays yellow forever, because the allocator refuses to put a replica on the same node as its primary — two copies on one machine buy no redundancy. The fix on a dev box is `number_of_replicas: 0`, not more debugging. ## Red Red means at least one primary shard is unassigned. Documents that hash to that shard are simply absent: searches over the index return partial results (with `_shards.failed` non-zero) and indexing requests routed to it fail. Red is *not* automatically permanent data loss — the usual cause is that the node holding that primary is down or restarting, and the shard comes back when it returns. It becomes real loss only when no copy of that shard exists anywhere any more. A subtlety worth saying out loud in an interview: red is per-shard, not per-cluster. The other shards of the same index, and every other index, keep serving traffic normally. "The cluster is red" is often mis-stated as "the cluster is down". ## How to look it up `GET _cluster/health` gives the colour plus counters: `active_shards`, `unassigned_shards`, `initializing_shards`, `relocating_shards`, `number_of_nodes`. Add `?level=indices` to get a colour per index, or `?level=shards` for per-shard detail. `GET _cat/indices?v&health=red` lists the offending indices. `GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason` lists the individual unassigned shards with a reason code such as `NODE_LEFT`, `INDEX_CREATED`, `ALLOCATION_FAILED`, or `CLUSTER_RECOVERED`. When the reason is not obvious, `GET _cluster/allocation/explain` names the rule that blocked the allocation. ## Typical causes - A data node left the cluster: replicas of its shards go unassigned (yellow), and any primary it held goes unassigned until a replica is promoted (briefly red if there was no replica). - More replicas requested than nodes available: `number_of_replicas: 2` on a two-node cluster is permanently yellow. - Disk watermarks: when a node crosses the high watermark the allocator stops putting shards there, and a cluster with nowhere left to place a copy stays yellow. - Allocation filtering or awareness rules that no node satisfies — for example an index required to sit on a node attribute that no longer exists. - Repeated allocation failures: after `index.allocation.max_retries` attempts the allocator gives up until you ask it to retry. - A restore or a fresh index during initialization is briefly red/yellow while shards initialize; that is normal and self-resolving. ## What the colour is good for As an alert, green-to-yellow is a redundancy alert and yellow-to-red is an availability alert. Both should be scoped by *how long*: transient yellow during a rolling restart is expected and healthy, while yellow for an hour means the allocator cannot place a copy and needs investigation. Alerting on any non-green colour without a duration window produces noise every time a node restarts.

  • A single-node Elasticsearch cluster is yellow and no node has failed. Why?
    The index was created with at least one replica, and Elasticsearch never allocates a replica onto the same node as its primary — two copies on one machine give no redundancy. With only one node, the replica has nowhere to go and stays unassigned. Set `number_of_replicas: 0` for a single-node development cluster, or add a node.
  • Does a red Elasticsearch cluster reject every search request?
    No. Red means at least one primary shard is unassigned; queries against other indices work normally, and a query against the affected index still returns hits from its healthy shards, with the failure reported in the response's `_shards.failed` count. You can require completeness instead by rejecting partial results, but the default is to return what is available.
  • Should you alert on any non-green Elasticsearch cluster health?
    Not on the colour alone. Rolling restarts, node upgrades and new-index initialization all produce short yellow periods. Alert on yellow sustained beyond a few minutes and on red almost immediately, and include `unassigned_shards` in the alert so the on-call engineer sees the scale of the problem.

Think of it as a spare-tyre check, not an engine check: green means every wheel plus a spare, yellow means all four wheels but a missing spare, red means you are driving on three wheels.

saying these in an interview costs you the question

  • Says yellow means the cluster is down or losing writes
  • Claims red always means permanently lost data
  • Treats green as proof of healthy performance and disk
  • Thinks health colour reflects CPU, latency, or heap
  • Does not know cluster colour is the worst index colour

context

open as a page

How do you use GET _cluster/allocation/explain to diagnose an unassigned Elasticsearch shard?

level: seniorimportance: must knowfreq 62%

basics

~20 s

Call it with the index, shard number and whether it is the primary; the response gives unassigned_info explaining why the shard became unassigned, and per-node decider decisions where each NO or THROTTLE names the exact rule blocking allocation on that node.

open as a page

What do the master, data, ingest, and coordinating-only node roles do in an Elasticsearch cluster?

level: middleimportance: should knowfreq 68%

basics

~20 s

In node.roles, master-eligible nodes can be elected to own the cluster state, data nodes hold shards and execute queries on them, ingest nodes run ingest pipelines, and a node with an empty roles list is coordinating-only: it routes requests and merges results.

open as a page

How do you restart an Elasticsearch data node for maintenance without triggering a full shard reallocation?

level: seniorimportance: should knowfreq 55%

basics

~20 s

Set cluster.routing.allocation.enable to primaries so replicas of the departing node are not rebuilt elsewhere, stop non-essential indexing and flush, restart the node, then set the setting back to all and wait for green. Delayed allocation covers short absences automatically.

open as a page

How does Elasticsearch's voting configuration decide whether a master can be elected after nodes leave?

level: seniorimportance: should knowfreq 48%

basics

~20 s

Elasticsearch maintains a voting configuration: the set of master-eligible nodes whose votes count. An election needs a strict majority of that set, so the cluster survives losing fewer than half of it. Below that it has no master and rejects cluster-state changes.

open as a page

How would you lay out an Elasticsearch cluster across three availability zones so losing one zone keeps it writable?

level: principalimportance: should knowfreq 38%

basics

~20 s

Put one master-eligible node in each zone so a majority survives losing one, spread data nodes evenly, tag every node with a zone attribute, enable allocation awareness on that attribute so shard copies land in different zones, and keep at least one replica per index.

open as a page