What does spark.sql.adaptive.coalescePartitions.enabled do to a stage's task count?

## The problem it solves Every shuffle in Spark SQL produces a fixed number of partitions, taken from `spark.sql.shuffle.partitions`, whose default is 200. That number is chosen before the query runs and applies to every shuffle in it. It is almost always wrong for at least one stage: a filtered branch that emits 300 MB gets 200 partitions of 1.5 MB each — 200 tasks whose scheduling and shuffle-fetch overhead dwarfs their work — while a stage that emits 2 TB gets 200 partitions of 10 GB each, which spill and often fail. Before Spark 3, tuning this by hand per job (and sometimes accepting a compromise between the stages inside one job) was a routine part of Spark work. ## What coalescing does Adaptive Query Execution breaks the physical plan at shuffle boundaries into *query stages*. When a map stage finishes, its `MapStatus` output carries the exact size of every shuffle partition — no estimation involved. The coalescing rule reads those sizes and, instead of scheduling one reducer task per shuffle partition, assigns each reducer task a **contiguous range** of partitions whose combined size approaches the target. Two hundred 1.5 MB partitions might be served by five reducer tasks reading 40 partitions each. Contiguity matters: a reducer task fetches a range of partition ids from every map output, so merging neighbours is a pure bookkeeping change on the read side. No data is re-hashed, no second shuffle is written, and the rows still land in a task that owns whole partitions — so grouping and join semantics are untouched. ## The configuration surface - `spark.sql.adaptive.enabled` — the umbrella switch, `true` by default since Spark 3.2. Nothing below works without it. - `spark.sql.adaptive.coalescePartitions.enabled` — `true` by default. - `spark.sql.adaptive.advisoryPartitionSizeInBytes` — the target size, 64 MB by default. It is also the target used when splitting skewed partitions. - `spark.sql.adaptive.coalescePartitions.parallelismFirst` — `true` by default. When true, Spark **ignores** the advisory size and only enforces the minimum size, so it merges just enough to eliminate trivially small partitions and keeps parallelism high. The documentation recommends setting it to `false` on a busy cluster, where many small tasks waste more than they gain. - `spark.sql.adaptive.coalescePartitions.minPartitionSize` — 1 MB by default; the floor honoured when the advisory size is being ignored. - `spark.sql.adaptive.coalescePartitions.initialPartitionNum` — the partition count the shuffle *starts* with; falls back to `spark.sql.shuffle.partitions` when unset. That `parallelismFirst` default surprises people: they set the advisory size to 128 MB, see tasks of a few megabytes, and conclude AQE is broken. It is doing exactly what it was configured to do. ## Why it only merges The map side has already written its output into N buckets by the time AQE looks. Splitting one of those buckets into finer key groups would require re-partitioning the data — another shuffle. Merging neighbours requires nothing. That asymmetry drives the recommended tuning pattern: pick a generous `initialPartitionNum` (thousands, for a large job), let the shuffle write fine-grained buckets, and let coalescing produce sensible reducer tasks per stage. You trade a little map-side bookkeeping for the ability to size every stage correctly, instead of one number for all of them. (The exception is skew, where AQE *can* split a partition — but only for joins, and by splitting the map-output blocks of one oversized partition, not by re-hashing keys.) ## What it does not fix Coalescing balances *small* partitions; it does nothing about one partition being enormous. If a hot key produces a 9 GB bucket, coalescing leaves it alone (merging it with a neighbour would only make it worse) and the straggler task remains. The skew-join rule, salting, or broadcasting is what addresses that. It also is not the DataFrame `coalesce(n)` transformation. That is a user-invoked narrow transformation that reduces partitions in the *current* stage without a shuffle, and, because it collapses the stage's parallelism upstream, it can starve the computation feeding it. AQE coalescing is an automatic runtime decision about *post-shuffle* reducer tasks. Sharing a name is unfortunate; interviewers ask about it precisely because the two are confused. ## Seeing it work In the SQL tab, an AQE plan is rooted at an `AdaptiveSparkPlan` node whose `isFinalPlan` flag flips to `true` once execution has settled, and the shuffle-read nodes report how many partitions were coalesced. The simpler signal is the stage's task count: fewer tasks than `spark.sql.shuffle.partitions`, with sensible per-task shuffle read sizes, means coalescing fired.