In Spark, which job patterns push work onto the driver and stall the application?

## What only the driver does A Spark application has exactly one driver and many executors. The driver hosts your `main`/notebook code and the `SparkSession`; it parses and optimizes the query, builds the DAG, cuts stages, tracks where every shuffle output lives via the `MapOutputTracker`, decides which task goes to which slot, receives task results and status updates, and serves the Spark UI. Executors do one thing: run tasks and cache blocks. That asymmetry is the whole diagnosis. Executors scale out; the driver does not. Any work that lands on the driver is single-machine work in the middle of a distributed job, and while it runs the entire cluster can be idle. ## Pattern 1: pulling results back `collect()` brings every row of the result into the driver's heap. So does `toPandas()` in PySpark, and `take(n)` with a large `n`, and iterating a DataFrame row by row in a Python loop. It is the most common way to turn a healthy job into a driver OOM, and it is guarded — imperfectly — by `spark.driver.maxResultSize` (default 1g), which aborts the job when the serialized results of a stage exceed the limit. Raising that setting to make the error go away is exactly the wrong move: it converts a clear failure into an unbounded heap. The fix is to keep the aggregation distributed. Replace `sum(r.amount for r in df.collect())` with `df.agg(sum("amount")).first()[0]`; replace a driver-side loop that writes files with a partitioned `write`; use `limit` before `collect` when you genuinely want a sample. ## Pattern 2: broadcasts A broadcast join is a good optimization, but the mechanics route through the driver: the small side is collected to the driver, serialized, and then distributed to executors. A table that is "small" by warehouse standards can still be hundreds of megabytes, and an automatic broadcast triggered by a bad size estimate on a compressed source is a classic surprise. Symptoms are a long pause before a stage starts, driver GC pressure, and occasionally a broadcast timeout. ## Pattern 3: planning and file listing Before any task launches, the driver must enumerate input files and prune partitions. Against an object store with a deeply partitioned layout and hundreds of thousands of small files, that listing is a driver-side, latency-bound operation that can take minutes with the cluster completely idle. The same applies to enormous query plans: a generated SQL statement with thousands of unioned branches or deeply nested views spends real time in Catalyst on the driver. The Jobs tab shows nothing running because nothing has been submitted yet. ## Pattern 4: too many tasks The driver launches every task, deserializes every task result, and processes every heartbeat and metric update. At a few hundred or a few thousand tasks per stage this is negligible. At a hundred thousand tiny tasks it is not: launch overhead and result handling become a measurable fraction of the stage, and the event queue behind the Spark UI listeners can fall behind and start dropping events. The tell is that per-task durations in the summary metrics are dominated by scheduler delay rather than executor compute time. ## Diagnosing it The fastest check is the Executors tab: if active tasks are zero and CPU is flat across every executor while wall-clock time passes, the driver is the one doing something. Then look at the driver: a thread dump (available from the UI) shows whether it is in Catalyst, in file listing, in broadcast serialization, or in GC. Driver GC time climbing while executor GC is quiet points at collected results or broadcasts. Long gaps *between* jobs point at planning or listing; long gaps *inside* a stage point at scheduling overhead. ## Structural fixes Keep results distributed; write to storage instead of collecting. Cap broadcasts deliberately rather than relying on estimates from compressed files. Compact small files and keep partition counts on tables sane, so listing is cheap. Coarsen partitioning when tasks are shorter than roughly a second, so each task does enough work to justify its scheduling cost. Where an interactive client is the problem, Spark Connect (GA in Spark 4.0, available from 3.4) separates the client from the driver so a heavy notebook process is no longer the same JVM that schedules the cluster. ## The interview framing Say it as a role statement first — one driver plans and schedules, many executors compute — then name the four patterns that violate it, then give the diagnostic (idle executors plus a busy driver) and the fix per pattern. That sequence demonstrates you have operated Spark rather than only written it.