In Spark, how does a reduce task learn where to fetch its shuffle blocks?

## The problem the fetch path solves After the map stage, one reduce partition's rows are scattered: a slice of it sits inside every map task's data file, on whichever node that task ran. Reduce task 57 must locate all those slices and stream them in. Spark solves this with a driver-side directory plus a client-side fetcher. ## Registration: MapStatus As each map task completes, it returns a `MapStatus` to the driver. It carries two things: the `BlockManagerId` (host, port and executor id) where the output lives, and the size of each reduce partition inside that task's data file. Sizes are stored compactly, and when a shuffle has a large number of partitions Spark switches to a `HighlyCompressedMapStatus` that records the exact sizes only of unusually large blocks and an average for the rest — otherwise the status objects themselves would bloat driver memory on a shuffle with tens of thousands of partitions. The driver's `MapOutputTrackerMaster` holds these statuses per shuffle id, which is why it is also the component that knows when map output has been *lost* and must be regenerated. ## Lookup: MapOutputTrackerWorker Every executor runs a `MapOutputTrackerWorker`. The first task on that executor needing shuffle X calls `getMapSizesByExecutorId`, which asks the driver for the serialized statuses, caches them locally, and returns the block ids grouped by the executor address that holds them, each with its expected size. Subsequent tasks on the same executor reuse the cache; the driver broadcasts large status maps rather than answering thousands of identical RPCs. This is one of the driver-side costs of very wide shuffles. ## Transfer: ShuffleBlockFetcherIterator The reduce task then builds a `ShuffleBlockFetcherIterator`, which splits requests into two classes: - **Local blocks** — output written by a map task that ran on this same executor. Read straight from disk with a seek into the data file, using the index file for the offset. No network involved. - **Remote blocks** — fetched over Netty from the peer executor's block manager, or from the node's external shuffle service when one is running. Requests are grouped by target address so many small blocks from one host travel together. Flow control matters here because a reduce task can be pulling from hundreds of peers at once. `spark.reducer.maxSizeInFlight` (default 48m) caps how much data may be outstanding, and the iterator issues several concurrent requests within that budget so a single slow node does not idle the reader. Companion limits bound requests and blocks in flight per remote address, which protects a hot node from being hammered by every reduce task at once. Blocks arrive out of order; the iterator hands them to the operator as they land, which is why shuffle reads are streamed and only aggregations or sorts that need all input buffer it. ## Where the fetch can fail Because the fetcher talks to a specific host and port that the tracker supplied, anything that invalidates that address surfaces here: the executor exited, the node was reclaimed, the disk holding the file is gone, or the shuffle service is saturated and times out. After a bounded number of retries the task raises a fetch failure, and the scheduler — not the task retry logic — takes over, because the fix is to regenerate the missing map output rather than to rerun the reader. ## Reading the metrics The stage's shuffle-read metrics decompose exactly along this path: remote bytes read, local bytes read, remote blocks fetched, and fetch wait time. A large fetch-wait time with modest bytes read points at many tiny blocks or a struggling peer, not at a slow computation. Local-versus-remote split tells you how much locality the scheduler actually achieved. ## What interviewers are checking Whether you can say who knows the locations (the driver), who does the reading (the reduce task's executor), and what limits the traffic. Candidates who believe reduce tasks broadcast a request to the cluster, or that the driver relays shuffle data, are describing a system that would not scale — and the misconception usually goes with an inability to explain fetch failures.