How would you size a new Hadoop cluster's DataNode storage, NameNode heap and worker node shape?

## Three budgets, not one number The common mistake is to size a Hadoop cluster from a single figure — "we have 300 TB". Storage, metadata and compute scale on different inputs and fail in different ways, so plan each one. ## Budget 1: raw storage Start with logical data: current volume plus projected growth over the planning horizon, after compression and after applying a retention policy (deciding what you *delete* is a capacity lever people forget). Multiply by the durability factor. Three-way replication means 3× raw for every logical byte. Hadoop 3's erasure coding is the alternative: an RS(6,3) scheme stores six data and three parity cells, giving roughly 1.5× overhead instead of 3× and tolerating three losses. It is not free — reconstruction after a failure is CPU- and network-heavy, and EC files carry functional restrictions — so it belongs on warm/cold data, not on the hot working set. Then add headroom. Do not plan to fill DataNodes: block placement, the balancer and the temporary spike of re-replication when a node fails all need slack, and jobs need space for output before old data is dropped. Planning to about 70–75% of raw is the usual discipline. Separately reserve space that HDFS must never claim with `dfs.datanode.du.reserved`, and remember that YARN's `yarn.nodemanager.local-dirs` (shuffle spill and container work dirs) and log dirs live on the same disks. So 300 TB logical at replication 3 is 900 TB of replicas, and you buy comfortably more than that — over a petabyte raw — not 900 TB exactly. ## Budget 2: NameNode metadata The NameNode holds the namespace in memory, and its heap is driven by **object count** — files, directories and blocks — not by how many bytes those files contain. A petabyte in a few thousand large files is trivial for the NameNode; the same petabyte in a hundred million tiny files may not fit in heap at all. Plan it empirically: run `hdfs oiv` over a representative `fsimage` to count objects, measure the heap your current NameNode actually uses, and project forward on the growth rate of *object count*. Rules of thumb about bytes per object circulate, but the honest engineering answer is to measure your own namespace rather than trust one. Set heap in `hadoop-env.sh` (`HADOOP_HEAPSIZE_MAX` or `HDFS_NAMENODE_OPTS`), and be aware that very large heaps bring long GC pauses — a pause that outlives the ZooKeeper session timeout will trigger a spurious HA failover. If object growth is the binding constraint, the levers are compaction of small files, HDFS federation, or moving cold data elsewhere. ## Budget 3: worker node shape Work backwards from the workload's container profile: how much memory a typical task needs, how many you want running concurrently, and whether the jobs are CPU-bound or scan-bound. - **Memory.** Node RAM minus the OS, the DataNode JVM and the NodeManager JVM is what YARN may hand out; put that figure in `yarn.nodemanager.resource.memory-mb`. It is not derived from physical RAM for you, and setting it to the full machine size is how you get OOM-killed daemons. - **Cores.** `yarn.nodemanager.resource.cpu-vcores` should reflect usable cores, allowing for the daemons and for the fact that vcores are a scheduling abstraction, not a cgroup guarantee unless you enable CPU enforcement. - **Disks.** Many independent disks, JBOD, each its own directory in `dfs.datanode.data.dir`. Do not RAID DataNode disks: HDFS already replicates across nodes, and RAID costs you both capacity and parallel spindle throughput. Sequential scan bandwidth per node is often the real ceiling on job runtime, so disk count matters as much as capacity per disk. - **Balance.** A node with huge capacity per disk but few spindles stores a lot and reads it slowly; that shows up as jobs that are slower on a "bigger" cluster. ## Master nodes and the network Master nodes are sized for reliability, not throughput: NameNode heap plus redundant metadata directories, three JournalNodes on separate machines, a ZooKeeper ensemble that is not competing with heavy I/O. The network matters too — shuffle-heavy workloads push a lot of traffic across rack uplinks, and an oversubscribed top-of-rack switch will cap the cluster long before the disks do. ## Multi-tenancy and validation If several teams share the cluster, capacity planning includes queue sizing in the scheduler and an explicit policy on what happens when the cluster is full: whose jobs wait. Decide that before the cluster is full, not during the first incident. Finally, validate rather than trust the spreadsheet. Run a representative workload on a small number of the proposed nodes, measure achieved scan throughput, container density and job runtime, and extrapolate from that. Then instrument the four early-warning signals — DataNode fill percentage, NameNode heap used and object count, queue wait time, and rack uplink utilisation — and re-plan when any of them trends toward its limit rather than when it hits it.