When would you still build a new platform on HDFS instead of object storage?

## The question behind the question This is a strategy question, not a trivia question. The interviewer wants to hear whether you can separate *what HDFS actually provides* from *the era it belongs to* — and whether you will give a defensible "almost never, except…" instead of either nostalgia or reflexive dismissal. ## What HDFS genuinely gives you **Data locality.** The NameNode knows which DataNodes hold each block, and schedulers use that: a Spark task can be placed on the machine already holding its input and read from local disk. On a well-provisioned on-prem cluster with modest network bandwidth, this is a real throughput advantage. **A real filesystem contract.** A hierarchical namespace, atomic directory rename, append and truncate. Anything that was written against filesystem semantics — the classic `FileOutputCommitter`, Hive's dynamic-partition insert, older ecosystem tools — works without modification. **Predictable low-latency reads on local disks.** HBase is the standing example: a random-read store that wants short, consistent I/O paths rather than per-request object-store latency. **It runs where there is no cloud.** Air-gapped defence and government environments, strict data-residency regimes, or a site with no object store at all. ## What it costs **Operational weight.** A production HDFS is not one service. Highly available deployments need an active and a standby NameNode, a JournalNode quorum for the shared edit log, and ZooKeeper plus failover controllers to arbitrate. Someone has to own upgrades, rebalancing, decommissioning and the occasional metadata recovery. **A namespace ceiling.** Every file, directory and block is an object in the NameNode's heap, so the metadata capacity of a cluster is bounded by one JVM's memory. Federation and router-based federation exist to shard the namespace, but they add topology rather than remove the constraint. **Coupled scaling.** Capacity comes from DataNodes, and DataNodes are machines you keep powered whether or not you are computing. You cannot grow a petabyte of cold storage without also growing the compute footprint. **Storage multiplier.** Default replication is three copies. Hadoop 3's erasure coding cuts that materially for cold data, at the cost of more expensive reconstruction on failure, but it is still local disk you own. ## The modern default, and what replaces each property A greenfield platform in 2026 is object storage (S3, ADLS, GCS; MinIO or Ceph on-prem) holding columnar files, with an open table format — Iceberg, Delta Lake or Hudi — layered on top, and a catalog for discovery. The table format is what replaces the property people actually relied on HDFS for: **atomic commit**. Instead of renaming a directory into place, the engine writes new data files and then atomically swaps a metadata pointer, which also brings snapshot isolation, time travel and schema evolution that HDFS never offered. Locality is replaced by aggregate bandwidth plus aggressive pruning: large files, partitioning, predicate and projection pushdown, and caching. Multiple engines can read the same tables concurrently without sharing a cluster. ## So when does HDFS still win Be concrete. On-premises and air-gapped, with no S3-compatible layer available. An existing Hadoop estate where the data is already there and migration cost dominates. A co-located HBase or similar latency-sensitive store. A workload whose economics genuinely favour owned hardware at sustained high utilisation. Notice that every one of these is an existing-context argument, not a design-from-scratch one — which is exactly the point. ## The migration angle If the answer is "move off", say how. Bulk-copy with `hadoop distcp` to an `s3a://` target, re-register tables in the new catalog, convert to a table format, and — the step people forget — fix the commit path, because a job that relied on rename-based commit will be slow or unsafe against object storage until it uses an S3A committer or a table format. Run both in parallel and cut over per dataset rather than in one move. One clarification worth volunteering: Spark's shuffle never used HDFS. Intermediate shuffle data goes to executor-local disks under `spark.local.dir`, so "we need HDFS for shuffle" is not a reason to keep it.