In YARN, what do the ResourceManager, NodeManager and ApplicationMaster each do?
answer
- three daemons, two of them cluster-wide
- one coordinator per job, not per cluster
- who allocates versus who executes
- the coordinator itself lives in a container
- the scheduler never watches a task
basics
~20 sYARN splits cluster management three ways: the ResourceManager schedules containers across the whole cluster, a NodeManager runs and monitors containers on each machine, and every application gets its own ApplicationMaster that requests containers and drives that job.
solid answer
~40 sYARN separates *cluster resource management* from *application logic*. The **ResourceManager** is the single cluster-wide daemon; its Scheduler decides which application gets which container out of the shared pool, and its ApplicationsManager accepts submissions and launches each application's first container. The **NodeManager** runs on every worker: it advertises that machine's memory and vcores, heartbeats to the ResourceManager, launches containers on request, monitors their resource usage, and kills ones that exceed their limits. The **ApplicationMaster** is per-application, not per-cluster — it runs inside an ordinary container, negotiates further containers from the ResourceManager through `allocate()` heartbeats, asks the relevant NodeManagers to start them, tracks progress, and re-requests containers after failures. The ResourceManager never watches individual tasks; that is entirely the ApplicationMaster's job.
code
bash · 4 linesyarn application -list -appStates RUNNING
yarn node -list -all
yarn application -status application_1699034221_0007
yarn logs -applicationId application_1699034221_0007go deeper
Be able to name the three components and say in one sentence what each does. Interviewers commonly open a Hadoop conversation here, and a confident allocate-versus-execute-versus-coordinate answer is enough.
Explain the split as a design decision: why per-application coordinators replaced the single Hadoop 1 JobTracker, and what a container actually is — a memory and vcore lease plus a command, not a Docker container.
Show you have operated this: what you look at when an application misbehaves, how the ApplicationMaster's own resource footprint counts against a queue, and what NodeManager auxiliary services such as the external shuffle service buy you.
Own the argument for why resource management is separated from application logic at all, and be able to compare that separation with how a container orchestrator divides the same responsibilities when deciding what a platform should run on.
## What problem YARN solves Before YARN, Hadoop's JobTracker did two unrelated jobs at once: it managed cluster resources *and* it tracked every map and reduce task of every job. That made it a scalability bottleneck and locked the cluster into one processing framework. YARN (Yet Another Resource Negotiator, introduced in Hadoop 2 and the model in Hadoop 3.x) splits those responsibilities into a cluster-wide resource broker plus a per-application coordinator, which is why Spark, Tez, Flink and MapReduce can all share one YARN cluster. ## The unit of allocation: a container A YARN **container** is a lease on a slice of one machine's resources — a memory amount and a number of vcores — plus a command line to run inside it. It is not a Docker container by default (though the LinuxContainerExecutor can be configured to use Docker). Everything a YARN application runs, including its own coordinator, runs inside a container. ## ResourceManager The ResourceManager is a single logical daemon (usually deployed as an HA pair with ZooKeeper-based failover). It has two internal pieces: - **Scheduler** — pure allocation. It matches container requests against free capacity according to a pluggable policy (`yarn.resourcemanager.scheduler.class`, Capacity Scheduler by default in Apache Hadoop 3). It does *no* monitoring, *no* task restarts, and offers no guarantees about what runs inside a container. - **ApplicationsManager** — accepts application submissions, validates them, and allocates and launches the first container of each application, the one that hosts the ApplicationMaster. It also restarts a failed ApplicationMaster within the configured attempt limit. The ResourceManager knows about applications and containers. It does not know that a Spark job has 4,000 tasks or that a MapReduce job is in its reduce phase. ## NodeManager One NodeManager per worker machine. Its responsibilities: - Advertise the machine's capacity — `yarn.nodemanager.resource.memory-mb` and `yarn.nodemanager.resource.cpu-vcores` — and heartbeat liveness and container status to the ResourceManager. - **Localize** resources: download the job's jars, archives and files from HDFS to the local disk before starting a container. - Launch containers through a ContainerExecutor, and monitor their memory and CPU use, killing a container that exceeds the memory it was granted. - Run **auxiliary services** that outlive individual containers — the MapReduce shuffle handler, and Spark's external shuffle service when it is enabled. - Aggregate container logs to HDFS when `yarn.log-aggregation-enable` is on, which is what makes `yarn logs -applicationId <id>` work after an application finishes. A NodeManager never decides *which* application gets resources. It only executes and polices what the ResourceManager granted. ## ApplicationMaster This is the piece people miss. Every submitted application gets its own ApplicationMaster, running in a container on some NodeManager chosen by the ResourceManager — never on the ResourceManager host, and (in cluster deploy mode) never on the client machine. It is framework-specific code: MapReduce ships `MRAppMaster`, Spark ships its own, and in Spark's cluster deploy mode the driver itself runs inside the ApplicationMaster container. The ApplicationMaster: 1. Registers with the ResourceManager. 2. Sends periodic `allocate()` heartbeats carrying resource requests (how many containers, how much memory and how many vcores each, and locality preferences such as "on this host" or "on this rack"). 3. Receives allocated containers in the heartbeat responses and contacts the owning NodeManagers to start processes in them. 4. Monitors its own tasks, re-requesting containers for the failed ones and applying its own retry and speculation policy. 5. Unregisters and exits when the application finishes, releasing its containers. Because the retry logic lives here, a task failure never touches the ResourceManager. Scaling the cluster scales the number of ApplicationMasters rather than the load on one central tracker. ## End-to-end submission A client submits an application to the ResourceManager. The ResourceManager picks a node with free capacity and asks that NodeManager to launch the ApplicationMaster container. The ApplicationMaster starts, registers, and begins asking for containers. As containers are granted, the ApplicationMaster tells the relevant NodeManagers to launch the actual work. The application passes through the states NEW, SUBMITTED, ACCEPTED (accepted by the scheduler, waiting for its ApplicationMaster container), RUNNING, and finally FINISHED, FAILED or KILLED. ## Vocabulary traps A YARN **container** is a resource lease, not a Docker image. A MapReduce **task** is a whole mapper or reducer, which occupies one container; a Spark **task** is one partition's work inside an executor, and one Spark executor occupies one YARN container and runs many tasks. And a YARN application's coordinator is the ApplicationMaster — a Spark driver and a YARN ApplicationMaster coincide only in cluster deploy mode.
- Where does the ApplicationMaster actually run, and why does that matter?In an ordinary container on some NodeManager, chosen by the ResourceManager — it is the application's first container. It matters because the ApplicationMaster consumes cluster resources like any other container, counts against the queue's ApplicationMaster budget, and dies with its node: if that machine is lost, the whole application attempt is lost and must be retried.
- What is the ResourceManager deliberately not responsible for?Task-level monitoring, retries, speculation and progress reporting. Its Scheduler only matches container requests to free capacity; it makes no guarantee about what runs inside a container or whether it succeeds. Pushing that work into per-application ApplicationMasters is precisely what removed the Hadoop 1 JobTracker bottleneck.
- What are NodeManager auxiliary services and why do they exist?Long-lived services hosted in the NodeManager process, outside any container, so their data survives the container that produced it. The MapReduce shuffle handler serves map output to reducers after the mapper's container exits, and Spark's external shuffle service does the same for executors — which is what allows executors to be removed by dynamic allocation without losing shuffle files.
The ResourceManager is a building's letting agent who only decides who gets which rooms; the NodeManager is the caretaker of one floor who unlocks doors and throws out tenants who overrun their space; the ApplicationMaster is the project lead each tenant sends in to actually run their work in the rooms they were given.
saying these in an interview costs you the question
- Says the ResourceManager launches and monitors every individual task
- Thinks one ApplicationMaster coordinates the whole cluster
- Claims the ApplicationMaster runs on the ResourceManager host
- Says the NodeManager decides which application gets resources
- Confuses YARN with HDFS and thinks YARN stores the data