In Hadoop, what do core-site.xml, hdfs-site.xml and yarn-site.xml each configure, and what overrides them?

## Four files, four scopes Hadoop configuration lives in XML files under `$HADOOP_CONF_DIR` (usually `etc/hadoop`), each a flat list of `<property><name>…</name><value>…</value></property>` entries. The split is by subsystem, not by node role — the same files are pushed to every machine: - **`core-site.xml`** — anything shared: `fs.defaultFS` (the default filesystem URI, e.g. `hdfs://prod`), `hadoop.tmp.dir`, `io.compression.codecs`, `ha.zookeeper.quorum`, `hadoop.security.authentication` and the proxyuser rules. - **`hdfs-site.xml`** — NameNode and DataNode: `dfs.replication`, `dfs.blocksize`, `dfs.namenode.name.dir`, `dfs.datanode.data.dir`, `dfs.namenode.handler.count`, and the whole HA block (`dfs.nameservices`, `dfs.ha.namenodes.*`, `dfs.namenode.shared.edits.dir`, fencing, the failover proxy provider). - **`yarn-site.xml`** — ResourceManager and NodeManagers: `yarn.resourcemanager.hostname` or the HA `rm-ids` set, `yarn.nodemanager.resource.memory-mb` and `.cpu-vcores`, `yarn.nodemanager.local-dirs` and `log-dirs`, `yarn.nodemanager.aux-services` (set to `mapreduce_shuffle` for MapReduce), and the scheduler class. - **`mapred-site.xml`** — the MapReduce framework: `mapreduce.framework.name=yarn`, `mapreduce.map.memory.mb`, `mapreduce.job.reduces`, JobHistoryServer addresses. Alongside them sit non-XML files: `hadoop-env.sh` (JVM-level environment such as `JAVA_HOME`, `HADOOP_HEAPSIZE_MAX`, `HDFS_NAMENODE_OPTS`), `workers` (host list used by the start scripts), `capacity-scheduler.xml`, and `log4j.properties`. ## The override chain Resolution order, weakest to strongest: 1. **`*-default.xml`**, bundled inside the Hadoop jars. You never edit these — they are the documented reference for what a property means and what its default is. 2. **`*-site.xml`** on the classpath, i.e. whatever is in `$HADOOP_CONF_DIR` on *that* machine. 3. **Per-application settings** — a `Configuration.set(...)` in code, `-D key=value` on the command line, or `-conf` pointing at another file. That last layer is why `<final>true</final>` exists. Marking a property final in a site file makes the daemon refuse a job's attempt to override it, and log a warning. Operators use it for things that must not be negotiable — security settings, `yarn.nodemanager.local-dirs`, memory caps. ## Client-side versus server-side properties This catches people out. Some properties are read by the **client** at the moment it acts: `dfs.replication` and `dfs.blocksize` are sent by the client when it creates a file, which is why two users on different edge nodes can write files with different replication into the same cluster, and why editing `dfs.replication` on the NameNode changes nothing about files that already exist (use `hdfs dfs -setrep` for those). Others are purely **server-side**: `dfs.namenode.name.dir` or `yarn.nodemanager.resource.memory-mb` are meaningless in a client's config and only take effect where the daemon reads them. ## Getting configuration onto the cluster Hadoop does not distribute its own configuration. Changing a property means shipping the file to every relevant host with your configuration-management tool and restarting the affected daemons — which is why a rolling restart plan matters on a live cluster. A handful of things can be refreshed without a restart: - `hdfs dfsadmin -refreshNodes` — re-reads the include/exclude host lists (used to decommission a DataNode). - `yarn rmadmin -refreshQueues` — re-reads `capacity-scheduler.xml`. - `hdfs dfsadmin -reconfig namenode <host:port> start` — applies the subset of properties marked reconfigurable at runtime. Everything else needs the daemon restarted, and a mismatched config across nodes (one worker still pointing at the old ResourceManager) produces some of the most confusing failures in Hadoop operations. ## Verifying what is actually in effect Never assume the file you edited is the file the daemon loaded. `hdfs getconf -confKey dfs.blocksize` prints the resolved value from the local client configuration. Each daemon's web UI exposes a `/conf` endpoint that dumps the effective configuration *and the file each value came from* — the fastest way to prove that your change reached the NameNode. `hadoop conftest` validates that the XML files parse. And remember that a job's own configuration is visible in the ApplicationMaster / JobHistory UI, which settles arguments about whether a value came from the cluster or from the submitting user. ## Common mistakes Editing `*-default.xml` instead of the site file; editing the site file on one node only; expecting `dfs.replication` to retro-apply; setting a per-node value like `yarn.nodemanager.resource.memory-mb` uniformly across heterogeneous hardware; and forgetting that `hadoop-env.sh` — not any XML file — is where daemon heap is set.