In Spark, how does repartition differ from coalesce, and when is coalesce the wrong choice?

## Two ways to change the partition count Both `repartition` and `coalesce` change how many partitions a DataFrame or RDD has, and that is where the resemblance stops. The difference is whether a shuffle happens, and a shuffle is what buys you the ability to move a row from any partition to any other. ## repartition: full shuffle, any count, even sizes `repartition(n)` writes every row to a shuffle file and reads it back on the target side. In the RDD API it is literally defined as `coalesce(n, shuffle = true)`. Because every row is redistributed, it can: - **increase** the partition count as well as decrease it, - produce **evenly sized** output partitions — with no columns given, Spark uses round-robin placement from a random starting offset, which balances rows regardless of how skewed the input partitions were. There are column-taking forms too. `repartition(n, "customer_id")` hash-partitions by that column, so all rows with the same value land in the same partition — this is what you want before a write partitioned by the same column, or before a `mapPartitions` that must see a whole group at once. Spark SQL's hash here is its own Murmur3-based `hash()` function applied modulo `n`, not the RDD `HashPartitioner`'s `key.hashCode`. `repartition("col")` with no number uses `spark.sql.shuffle.partitions` (200 by default). The cost is real: serialization, disk writes on the map side, network fetches on the reduce side, and a stage boundary. You pay it deliberately. ## coalesce: narrow merge, downward only `coalesce(n)` builds a new partition list where each output partition is the concatenation of several input partitions, preferring inputs already co-located on the same executor. No data crosses the network, no shuffle files are written, no stage boundary is created. It is essentially free. Two limitations follow from that design: 1. **It cannot increase the count.** `coalesce(200)` on a 50-partition DataFrame silently returns 50 partitions. There is no error; the request is simply ignored, because raising the count would require redistributing rows, which requires a shuffle. (`coalesce(n, shuffle = true)` opts back into one, and is just `repartition`.) 2. **It cannot rebalance.** If one input partition holds 90% of the rows, the merged partition containing it still holds 90% of the rows. ## The upstream-propagation trap This is the part interviews are really probing. Because `coalesce` introduces no stage boundary, the reduced parallelism applies to the *entire narrow chain above it* in the same stage. Consider: ```python (spark.read.parquet("s3://lake/events/") # 2,000 partitions .filter("country = 'PT'") .coalesce(1) .write.parquet("s3://lake/out/")) ``` A reasonable reader expects 2,000 tasks to scan and filter, then one task to write. What actually happens is one task that scans all 2,000 files, filters them, and writes — because scan, filter and write are all narrow and therefore fused into a single stage whose parallelism `coalesce(1)` has set to one. On a large input this turns a five-minute job into an hours-long one, or into an executor OOM. The fix is `repartition(1)`: the shuffle boundary lets the scan and filter run with full parallelism, and only the post-shuffle side runs single-threaded. You pay a shuffle to buy back the upstream parallelism, and on a heavily-filtered dataset that is a bargain. ## When each one is right Use **coalesce** when a stage has already produced many small, roughly balanced partitions and you just want fewer output files — classically after a highly selective filter, reducing 2,000 near-empty partitions to 50 — and when the reduction is modest so the upstream parallelism cap still leaves enough tasks. Use **repartition** when you need to *increase* parallelism, when partitions are unevenly sized, when you must group rows by a key before a write or a partition-local computation, or when the target count is drastically lower than the upstream count and you cannot afford to throttle the upstream stage. ## Interactions worth knowing Adaptive Query Execution (on by default since Spark 3.2) already coalesces over-provisioned *post-shuffle* partitions using runtime statistics, so a lot of the manual `coalesce`-after-`groupBy` tuning that older guides recommend is now unnecessary. Note also that an explicit `repartition(n)` historically blocked AQE's coalescing of that exchange, since you asked for a specific number on purpose; `spark.sql.adaptive.optimizeSkewsInRebalancePartitions` and the `REBALANCE` hint exist for the "give me evenly sized output, you pick the count" case. Finally, both operations are about *in-memory* partitions and therefore about task parallelism and output file count. Neither is related to `df.write.partitionBy(...)`, which creates directories in the output path.