Why does the HDFS NameNode struggle with ten million tiny files?

## The unit of cost is the object, not the byte HDFS's single-master design puts the whole namespace in the NameNode's JVM heap: every directory, every file, and every block is a live object. The NameNode does not care how much data a block holds — a block containing 20 KB occupies the same slot in the block map as one containing 128 MB. A common planning rule of thumb is on the order of a couple of hundred bytes of heap per namespace object; the exact figure depends on path length, replication factor and Hadoop version, but the shape is what matters: **cost scales with object count, not data volume**. So compare two ways of storing 200 GB: - 1,600 files of 128 MB → roughly 1,600 file objects and 1,600 block objects. - 10,000,000 files of 20 KB → 10,000,000 file objects and 10,000,000 block objects, plus whatever directory objects the layout adds. The second costs thousands of times more master memory for the same bytes on disk. Note also that disk usage is *not* the problem — HDFS blocks are logical, so a 20 KB file occupies 20 KB per replica. The waste is entirely in the master and in per-file overhead. ## What it actually breaks **Heap and garbage collection.** As the object count climbs, the NameNode's live set grows into tens of gigabytes. Old-generation collections on a heap that size produce pauses long enough that DataNodes miss heartbeat deadlines and clients time out. A NameNode that pauses is a cluster that pauses, because every open and every directory listing goes through it. **Restart time.** A cold start loads the fsimage (proportional to object count), replays the edit log, and then waits in **safe mode** until enough blocks have been reported by DataNodes at minimum replication. Both phases scale with block count. On a namespace bloated with small files, a planned restart turns from minutes into a long, visible outage. **RPC pressure.** Listing a directory of a million entries, or opening a million files, means a million round trips to the NameNode. Ingest jobs that create files one per record are the worst offenders: they hammer the master with create, addBlock and complete calls, and the edit log grows accordingly. **Downstream parallelism.** Processing frameworks derive splits from files and blocks. One tiny file usually yields one input split, so a job over ten million files launches on the order of ten million tasks, each reading 20 KB and each paying full scheduling, JVM or task-slot, and shuffle-registration overhead. Wall-clock time becomes almost entirely scheduling, and the job's driver or ApplicationMaster may itself run out of memory tracking that many tasks. **Columnar formats get worse, not better.** A Parquet or ORC file has a footer with per-column statistics and an internal row-group structure sized for hundreds of megabytes. Writing 20 KB Parquet files means the metadata dominates the payload, compression barely works because dictionaries have nothing to learn from, and predicate pushdown loses its value because every file is opened anyway. ## How small files get created Almost always by an ingest pattern nobody revisited: - a streaming sink writing one file per micro-batch per partition per output task, every minute, for a year - an over-partitioned write layout — partitioning by hour and by a high-cardinality key at once - a job whose output task count equals its input task count, so a wide upstream stage produces thousands of small output files - per-event or per-record file creation from an application pushing directly to HDFS ## Fixing it **Compact, and compact at the source.** The durable fix is to write fewer, larger files: coalesce output tasks before writing, buffer streaming output to a target file size rather than a fixed interval, and run a scheduled compaction job that rewrites yesterday's partition into a handful of large files. Target file sizes in the hundreds of megabytes, aligned with the block size. **Container formats.** `SequenceFile` and Avro container files pack many small records into one large file with a key per record. A Hadoop Archive (`hadoop archive`, producing a `.har`) bundles many files into one, collapsing the namespace cost while keeping the originals addressable — useful for cold data, awkward for active processing because it is read-only and adds an indirection. **Use a store designed for small records.** If the access pattern is genuinely "fetch one small object by key", HBase on top of HDFS is the right tool: it stores millions of small rows inside a modest number of large HDFS files. **Federation, as a scaling tool rather than a fix.** HDFS Federation runs multiple independent NameNodes, each owning a slice of the namespace over a shared pool of DataNodes, so the object ceiling multiplies. It raises the ceiling; it does not make ten million tiny files a good idea, and it adds operational complexity. **A bigger heap is a stopgap.** Raising the NameNode heap buys time and makes garbage collection pauses worse. It should be a deliberate breathing-room measure while compaction is built, never the answer. ## Diagnosing it `hdfs dfs -count` on a path reports directory, file and byte counts — divide bytes by files to get the average size. `hdfs fsck` on a path reports total blocks and average block size, which is the single most damning number: an average block size in the kilobytes on a multi-terabyte tree tells the whole story. NameNode JMX exposes the total file and block counts and heap usage, and those are the metrics to alarm on, well before the heap is full.