In Spark, what makes a transformation narrow or wide, and why does the difference matter?

## Dependencies, not operation names The words *narrow* and *wide* describe a relationship between a child RDD's partitions and its parent's partitions, not an intrinsic property of a function name. - **Narrow dependency:** each partition of the child depends on at most a small, fixed number of parent partitions. In the common case it is exactly one. - **Wide (shuffle) dependency:** a partition of the child may depend on rows from *all* parent partitions, because the rows it needs are scattered by whatever criterion the operation groups on. That single distinction drives most of Spark's execution behaviour. ## Which operations fall where Narrow: `map`, `mapPartitions`, `flatMap`, `filter`, `sample` (without replacement bookkeeping), `union`, `coalesce`, and — importantly — a `join` whose two sides are **already co-partitioned** by the join key with the same partitioner. Wide: `repartition`, `repartitionByRange`, `groupByKey`, `groupBy`, `reduceByKey`, `aggregateByKey`, `distinct`, `sortByKey`, `orderBy`, and any `join` or set operation whose inputs are not already partitioned on the key. The conditional cases are the interesting ones. `join` is wide *by default* and narrow *when Spark can prove co-partitioning* — the same reason `equals` on a custom `Partitioner` matters. A broadcast join is a third thing again: it avoids the shuffle by sending the small side to every executor, making the join narrow with respect to the large side. ## Why it matters — four consequences **1. Pipelining and stage boundaries.** Spark fuses consecutive narrow transformations into one stage and runs them within a single task, one row (or one column batch) at a time, without materialising intermediate results. `df.filter(...).select(...).withColumn(...)` costs one pass. A wide transformation cannot be fused: the parent side must finish and write its output before the child side can fetch it, so it becomes a stage boundary. **2. Cost.** A narrow chain touches no network and no shuffle disk. A wide transformation serialises rows, writes them to local disk, and pulls them across the network — usually the dominant cost of a job and the thing you tune first. **3. Partition count.** Narrow transformations preserve the partition count (except `coalesce`, which merges, and `union`, which sums). Wide transformations set it to `spark.sql.shuffle.partitions`, 200 by default, subject to adaptive coalescing at runtime. **4. Failure recovery.** Lineage recovery is cheap across narrow dependencies: lose one partition, recompute exactly one parent partition. Across a wide dependency it is expensive — the lost partition's inputs came from every parent, so unless the shuffle files survive, Spark must re-run the whole upstream stage. This is precisely why Spark materialises shuffle output to disk and why an external shuffle service exists. ## The `coalesce` special case `coalesce` is narrow, which sounds like unambiguously good news and is actually the source of its famous trap. Because it introduces no stage boundary, its reduced parallelism applies to everything fused with it — `read.filter().coalesce(1).write` runs the read and the filter with one task. Narrowness is what makes it cheap and what makes it dangerous. ## Reducing the shuffle rather than avoiding it You cannot make a genuine regrouping narrow, but you can make it cheaper. `reduceByKey` and `groupByKey` are both wide, yet `reduceByKey` applies the reduction **map-side** before writing shuffle files, so far fewer bytes cross the network; `groupByKey` ships every value. The DataFrame API does this automatically for standard aggregations, which is one concrete reason to prefer it over hand-written RDD code. Broadcasting a small dimension table removes the shuffle on the fact side entirely. Reusing an existing partitioning — for instance, joining twice on the same key without an intervening repartition — lets Spark skip the second exchange. ## Reading it off a plan In a physical plan, every wide dependency appears as an `Exchange` node (`Exchange hashpartitioning(user_id, 200)`), and in the Spark UI each `Exchange` is a stage boundary with shuffle read/write byte counts attached. Counting `Exchange` nodes in `df.explain()` is the fastest way to answer "how many shuffles does my job do", and cutting that count is usually the highest-leverage optimisation available. ## Where the vocabulary trips people A Spark *stage* is a set of tasks with no shuffle between them; the shuffle boundary defines it. That is unrelated to a MapReduce job's fixed map and reduce phases, even though a single Spark shuffle resembles one map/reduce pair — Spark simply chains as many as the query needs, keeping intermediate results in memory where it can, rather than round-tripping through a distributed filesystem between every pair.