How do you restart an Elasticsearch data node for maintenance without triggering a full shard reallocation?

## What happens when a node leaves The master notices the node is gone, marks its shards unassigned, and — for any primary it held — promotes an in-sync replica elsewhere so the index stays writable. Then the allocator wants to restore the configured replica count, which means building fresh copies on the remaining nodes. For a node that is genuinely dead, that is exactly right. For a node you are rebooting for a kernel patch, it is a large, pointless data movement that will be undone minutes later when the node returns and its own copies are found to be redundant. ## Delayed allocation `index.unassigned.node_left.delayed_timeout` is a per-index dynamic setting, default `1m`, that applies specifically to shards left unassigned because their node left. During the delay, the shards show as `unassigned` with `unassigned.reason: NODE_LEFT` and the cluster is yellow, but nothing is copied. If the node rejoins inside the window, it brings its shard copies with it and recovery is cheap: because replicas track the primary's operation history through retention leases, a returning replica usually replays only the operations it missed rather than copying whole segment files. Raising this to, say, `5m` across your indices before a maintenance window is the least invasive way to make short restarts free: ``` PUT _all/_settings { "settings": { "index.unassigned.node_left.delayed_timeout": "5m" } } ``` Note what it does **not** do: it never delays the promotion of a replica to primary. Data stays available immediately; only the rebuilding of redundancy is deferred. ## The rolling restart recipe The conventional sequence, one node at a time: 1. `PUT _cluster/settings` with `"persistent": { "cluster.routing.allocation.enable": "primaries" }`. While this is set, the allocator will allocate primary shards only — so replicas of the node you are about to stop are not rebuilt anywhere. 2. Stop non-essential indexing if you can, and run `POST /_flush` so that recent operations are committed to disk. That shortens recovery when the node comes back. 3. Stop the node, do the maintenance, start it again, and wait for it to rejoin (`GET _cat/nodes`). 4. Set `cluster.routing.allocation.enable` back to `all`. 5. Wait for the cluster to return to green before touching the next node. Moving on while still yellow stacks unassigned shards from two nodes and can turn the cluster red. The values for that setting are `all`, `primaries`, `new_primaries` and `none`; `all` is the default and the state you must return to. Leaving it at `primaries` after the window is a classic incident: the cluster looks fine but never restores replicas, and stays quietly yellow until someone notices. Managed Elasticsearch platforms drive this through the node shutdown API, which signals whether a node is going away temporarily or permanently. Elastic documents that API as intended for use by their orchestration products rather than by hand, so the settings-based recipe above remains the one to describe in an interview. ## Restart versus decommission The two operations want opposite behaviour, and conflating them is the mistake to avoid: - **Restart** (node returns): suppress reallocation. Delayed allocation plus `enable: primaries`. - **Decommission** (node never returns): *cause* reallocation, in a controlled way, before you power it off. Set `cluster.routing.allocation.exclude._name: "es-data-7"` at the cluster level, watch `GET _cat/allocation?v` until that node holds zero shards, then shut it down. The cluster never goes yellow, because every copy was moved while the source was still alive. Afterwards, clear the exclusion — a forgotten exclusion is one of the most common causes of a mysterious `filter` NO in allocation explain later. ## Recovery cost and throttles Recovery is deliberately rate-limited so that rebuilding shards cannot starve live traffic. `indices.recovery.max_bytes_per_sec` caps the byte rate per node (40mb by default on ordinary nodes in current versions, with higher defaults on dedicated cold and frozen tiers). `cluster.routing.allocation.node_concurrent_recoveries` limits how many recoveries a node participates in at once (default 2), with separate incoming and outgoing variants, and `cluster.routing.allocation.node_initial_primaries_recoveries` (default 4) governs the burst when a whole cluster restarts. On fast local disks and a fat network these defaults are conservative and are often raised temporarily to shorten a maintenance window — then put back, because leaving them high means the next unplanned node loss saturates the cluster. ## Pitfalls worth naming Restarting several nodes at once without waiting for green. Forgetting to re-enable allocation. Leaving a decommission exclusion in place. Setting `delayed_timeout` so high that a genuinely dead node's redundancy is never restored. And using a restart as a fix for unassigned shards — if the allocator is refusing to place a shard, restarting nodes adds recoveries to a cluster that already cannot place copies.