How does HDFS NameNode high availability keep a standby ready and stop two NameNodes from both writing?

## What HA has to solve Without HA, the NameNode is a single point of failure for the whole platform: it holds the only copy of the live namespace in memory, and if it dies every HDFS read and write, and therefore every job, Hive query and HBase region, stops until an operator restarts it and it finishes loading `fsimage`, replaying edits and collecting block reports — which on a large cluster is tens of minutes. HA replaces that with a second NameNode that can take over in seconds. ## The shared edit log: Quorum Journal Manager Every namespace mutation (create, rename, delete, block allocation) is appended to an edit log before it is acknowledged. In an HA cluster that log does not live on one disk; it lives on a set of `JournalNode` daemons — an odd number, in practice three — configured as: ``` dfs.namenode.shared.edits.dir = qjournal://jn1:8485;jn2:8485;jn3:8485/prod ``` The active NameNode writes each edit batch to all of them and considers it durable once a **majority** acknowledge. Three JournalNodes therefore tolerate one failure, five tolerate two. The critical rule is what happens when the majority is *not* reachable: the active NameNode aborts itself rather than accept writes it cannot durably record. Losing JournalNode quorum is an outage, not a silent degradation — which is why JournalNodes go on three separate machines with their own local disk for `dfs.journalnode.edits.dir`. ## Why the standby is hot, not cold The standby does two things continuously. It **tails the journal**, applying every edit to its own in-memory namespace, so its view of the directory tree is seconds behind the active. And it receives **DataNode heartbeats and block reports**: DataNodes are configured with the nameservice, not a single NameNode address, and report to every NameNode in it. That second point is what makes fast failover possible at all, because the block map — which node holds which replica — is never written to the edit log. A NameNode that has not been receiving block reports cannot serve reads until the cluster has reported in, which is the multi-minute startup cost HA is designed to avoid. ## Automatic failover: ZooKeeper and the ZKFC Set `dfs.ha.automatic-failover.enabled=true` and point `ha.zookeeper.quorum` (in `core-site.xml`) at a ZooKeeper ensemble. Each NameNode host then runs a `DFSZKFailoverController` (started as `hdfs --daemon start zkfc`, initialised once with `hdfs zkfc -formatZK`). The ZKFC health-checks its local NameNode over RPC and, if healthy, tries to hold an ephemeral znode representing "active". If the NameNode becomes unhealthy or the host dies, the session expires, the znode disappears, the peer ZKFC wins the election and transitions its NameNode to active. ## Fencing and split brain The scary case is an active NameNode that is *not* dead — just GC-paused, or partitioned — while a second one is promoted. With the Quorum Journal Manager, the journal itself resolves this: each writer takes an increasing **epoch** number when it becomes active, and JournalNodes reject writes from any lower epoch. A stale NameNode therefore cannot corrupt the namespace; its next write fails and it shuts down. Fencing methods configured in `dfs.ha.fencing.methods` — `sshfence` (SSH in and kill the process) or `shell(...)` — add belt and braces, mainly to stop the old process from answering stale *reads* to clients that have not failed over yet. Many QJM deployments configure `shell(/bin/true)` and rely on epoch fencing; if you use `sshfence` you must ensure the key is present and reachable, or a failover can hang waiting to fence a dead host. ## Clients Clients no longer address a host. They use `hdfs://prod/path`, and `dfs.client.failover.proxy.provider.prod` set to `ConfiguredFailoverProxyProvider` makes the client try each NameNode in `dfs.ha.namenodes.prod` and retry on a `StandbyException`. A running job survives a failover as a retry, not a failure. ## Beyond two NameNodes Hadoop 3 supports more than one standby, so you can tolerate two NameNode failures. Later Hadoop 3 releases also add the **Observer NameNode**, a standby that can serve consistent reads (clients issue an `msync` to bound staleness), which offloads read RPC from the active on very large clusters. Both are worth naming; neither is common on a small cluster. ## Operating it `hdfs haadmin -getServiceState nn1` reports active/standby. `hdfs haadmin -failover nn1 nn2` performs a graceful failover for planned maintenance. `-transitionToActive --forcemanual` exists but is dangerous and only valid when automatic failover is disabled. The alerts that matter: JournalNode quorum health, edit-log lag on the standby, ZKFC liveness, and the state where *neither* NameNode is active — that is a ZooKeeper or ZKFC problem, not a NameNode one.