How does an HDFS client write one block through a DataNode replication pipeline?

## Why a pipeline and not a fan-out HDFS writes every block to several DataNodes. The naive design would have the client send the block three times, once per replica — which triples the client's outbound bandwidth and makes the client the bottleneck. Instead HDFS chains the DataNodes: the client sends each packet **once**, to the head of a pipeline, and each DataNode forwards to the next as it receives. Every link carries roughly one block's worth of traffic, the client's uplink carries exactly one, and the cost is spread across the cluster fabric. ## Step by step **1. Create the file.** The client calls `create()` on the `DistributedFileSystem`. The NameNode checks that the path does not already exist and that the caller has permission, then records a new, empty file in the namespace and grants the client a **lease** — HDFS allows exactly one writer per file, and the lease must be renewed or the write is reclaimed. **2. Allocate a block.** The client writes into a `DFSOutputStream`, which buffers bytes locally. When a new block is needed the client asks the NameNode to allocate one; the NameNode returns a fresh block ID and an ordered list of target DataNodes chosen by the replica placement policy. **3. Build the pipeline.** The client opens a connection to the first DataNode, which connects to the second, which connects to the third. Setup acknowledgements flow back, and the pipeline is live. **4. Stream packets.** The stream splits data into packets (64 KB by default), each carrying checksums for its chunks. Packets are appended to the **data queue** and sent to the head DataNode. Each DataNode writes the packet to its local block file and forwards it downstream in the same motion, so the three writes overlap in time rather than happening one after another. **5. Acknowledge.** Every sent packet moves to the **ack queue**. The tail DataNode acknowledges to the middle, which acknowledges to the head, which acknowledges to the client. Only when a packet is acknowledged by every node in the pipeline is it removed from the ack queue — which is precisely what makes recovery possible. **6. Close.** After the last packet, the client flushes the queues and calls `complete()` on the NameNode. The file is complete once the NameNode sees at least `dfs.namenode.replication.min` replicas — one by default — for every block. Bringing the block up to the full replication factor happens afterwards, driven by the NameNode. When a block fills, the pipeline for it is closed and the whole allocate-and-build sequence repeats for the next block, generally with a different set of DataNodes. ## What happens when a DataNode in the pipeline dies This is the part interviewers push on, because it explains why the ack queue exists: 1. The pipeline is closed. 2. Packets still sitting in the ack queue — sent but not fully acknowledged — are pushed back to the **front** of the data queue, so no data is lost. 3. The surviving DataNodes' partial replica gets a new **generation stamp**, which lets the NameNode recognise the stale replica on the failed node as obsolete and schedule it for deletion when that node returns. 4. The failed node is removed from the pipeline and writing continues to the remaining nodes. Depending on policy the client may add a replacement node to restore the pipeline's width. 5. The NameNode later notices the block is under-replicated and instructs a DataNode to copy it to a fresh machine. The write only fails outright if the pipeline cannot keep the configured minimum number of replicas. ## Visibility and durability semantics HDFS is **write-once, read-many with append**: a file has a single writer, there is no random overwrite in place, and only appending to the end is supported. While a block is being written, other readers cannot generally see the newest bytes. `hflush()` pushes buffered data to all DataNodes in the pipeline and makes it visible to new readers, but does not guarantee it has reached disk; `hsync()` additionally forces the DataNodes to sync to their storage. Closing the file does both and finalises the block. That distinction is what bites people who use HDFS as a streaming sink: an unclosed file can appear to have zero length to a reader, and a client that dies without closing leaves a file whose last block is recovered through **lease recovery**, truncating to the last consistently acknowledged length. ## Where the client sits If the writing client is itself running on a DataNode — the usual case for a job task — the placement policy puts the first replica on that same machine, so the head of the pipeline is a local write with no network hop at all. A client outside the cluster gets a randomly chosen (and not overloaded) DataNode as the head instead, and pays a network hop for every byte.