In Spark, what determines where the DAG scheduler places a stage boundary?

## The rule in one line Spark cuts a stage wherever it cannot keep computing a partition locally. Formally, a boundary appears at every **wide dependency** (`ShuffleDependency`); everything else is fused into the current stage. ## Narrow versus wide A **narrow dependency** means each output partition draws from exactly one input partition. `map`, `filter`, `select`, `withColumn`, `flatMap`, `mapPartitions`, `union` and a broadcast (map-side) join are all narrow: the executor already holds everything a given output partition needs. Spark therefore never materializes the intermediate result; it composes the operators into one function evaluated record by record inside a single task. Ten chained narrow transformations still cost one stage. A **wide dependency** means an output partition can draw from many input partitions, because the row's destination is decided by its key rather than by where it already sits. Grouping, deduplication, key-based joins, global ordering and explicit repartitioning are all wide. To satisfy them, every executor must write records out bucketed by destination and every downstream task must fetch its bucket from every upstream task. That redistribution cannot be pipelined, so Spark ends the stage there. ## What the boundary actually costs A stage boundary is not just a bookkeeping line. The upstream `ShuffleMapStage` **materializes**: each of its tasks writes its output to local disk as shuffle files before the downstream stage begins. That has three consequences interviewers care about. First, it serializes execution — the downstream stage cannot start until enough upstream tasks have written their output, so a boundary is a barrier in the timeline. Second, it converts an in-memory pipeline into disk I/O plus network transfer, which is usually the dominant cost in a Spark job. Third, it creates a recovery point: because the map output exists on disk, a failed downstream task can simply re-fetch rather than recompute the whole lineage — and, conversely, losing an executor that holds map output forces Spark to recompute the parent stage. ## Reading boundaries out of a plan For DataFrames and Spark SQL, `df.explain()` is authoritative. Each `Exchange` node in the physical plan is exactly one stage boundary, and its partitioning tells you which kind: `hashpartitioning(key, n)` for a grouping or key join, `RoundRobinPartitioning(n)` for a bare `repartition(n)`, `rangepartitioning` for a global `orderBy`. Count the `Exchange` nodes, add one, and you have the stage count for that job. For RDDs the equivalent signal is `toDebugString`, where each indentation shift marks a `ShuffledRDD`. ## Boundaries you can remove Because boundaries dominate cost, the useful engineering move is deleting them rather than tuning them. A join whose smaller side fits in memory can be turned into a broadcast join, which is narrow — the small table is shipped to every executor and no exchange is needed at all. Two aggregations on the same key can share one exchange if you group once instead of twice. `coalesce(n)` narrows partition count without a shuffle, while `repartition(n)` always adds one, so the wrong choice at the end of a job inserts a boundary purely to write files. Reading data already bucketed on the join key can let Spark skip the exchange entirely. And an accidental `distinct()` or a `dropDuplicates()` inside a loop is a boundary someone added without noticing. ## What is not a boundary A source scan is not a boundary — partitioning at the read is decided by input splits, not by a shuffle. A `cache()`/`persist()` is not a boundary either; it changes where a later job reads from but does not cut the current stage. Adding executors does not add or remove stages. And an action does not create a boundary, it creates a job whose last stage is the `ResultStage`. ## Where adaptive execution fits From Spark 3.2 onwards adaptive query execution is enabled by default, and it re-plans *at* these boundaries: after a map stage completes, Spark has real output statistics and may coalesce the downstream partition count or switch a join strategy before launching the next stage. That means the physical plan you printed before running (`isFinalPlan=false`) can differ from what actually ran — but the boundary positions themselves still come from the same wide-dependency rule. ## The interview answer Say the rule (wide dependency), name three wide and three narrow operators, point at `Exchange` in the plan as the observable marker, and close with the consequence: boundaries force materialization to disk plus a network fetch, so the fastest job is the one with the fewest of them.