In Spark, how does HashPartitioner assign a key, and when do you want RangePartitioner instead?

## The partitioner's job A partitioner answers one question for a key-value dataset: *given this key, which partition does the row go to?* In the RDD API it is an actual object — the abstract class `Partitioner` with two members, `numPartitions` and `getPartition(key: Any): Int` — attached to a pair RDD and consulted by every shuffle. Spark ships two implementations, and Spark SQL has direct analogues of both. ## HashPartitioner `HashPartitioner(n).getPartition(key)` is `nonNegativeMod(key.hashCode, n)` — take the key's `hashCode`, force it non-negative, take it modulo the partition count. `null` maps to partition 0. What this buys you is the property every shuffle-based operator depends on: **equal keys go to the same partition**. `reduceByKey`, `groupByKey`, `join` and `distinct` all need every occurrence of a key to be gathered in one place, and hash partitioning is the cheapest way to guarantee it without knowing anything about the data. What it costs you is size control. Partition sizes are whatever the key distribution makes them. If 40% of your rows carry `user_id = 0` (the classic "unknown" sentinel), 40% of your data lands in one partition, one task runs for an hour while 199 finish in seconds, and that task may spill or die. Diagnosing and fixing that is the business of skew handling and adaptive execution. A second, subtler cost is hash quality. `HashPartitioner` uses the key's own `hashCode`, so a custom key class with a poor `hashCode`, or numeric keys that are all multiples of the partition count, can collapse onto a few partitions even when the data is perfectly balanced. In Spark SQL the equivalent is `HashPartitioning`, which applies Spark's own Murmur3-based `hash()` expression modulo the partition count rather than the JVM `hashCode`, and is generally better behaved. ## RangePartitioner `RangePartitioner` takes a different approach: sample first, then divide. It runs reservoir sampling over the parent partitions to estimate the key distribution, computes `n - 1` boundary keys, and then assigns each key by binary search into its range. Keys must have an ordering. Two properties follow: - **Balanced record counts.** Because boundaries are chosen from the observed distribution, a dense region of the key space is split across several partitions. Range partitioning tolerates skewed *distributions of distinct keys* far better than hashing does. (It cannot help with a single key that holds 40% of rows — that key belongs to exactly one range by definition.) - **Global order.** Partition i holds keys strictly less than partition i+1, so concatenating the partitions in order yields sorted output. This is why `sortByKey` and a DataFrame global `ORDER BY` use range partitioning: sorting within each partition then becomes enough. The costs are an **extra job** to draw the sample before the real shuffle can be planned, and **non-determinism**: the sample differs between runs, so the boundaries — and therefore which rows land in which output file — differ too. That surprises people who expect `repartitionByRange` to be reproducible, and it matters if a downstream system keys off file contents. ## The DataFrame equivalents You rarely construct a `Partitioner` in modern Spark, because the DataFrame API does not accept one; Catalyst chooses the physical distribution itself. What you do have: - `df.repartition(n, col("k"))` — hash-partition by `k` into `n` partitions. - `df.repartitionByRange(n, col("k"))` — range-partition by `k`, sampling to pick the bounds. - `df.orderBy(col("k"))` — a global sort, which plans a range-partitioning exchange followed by a per-partition sort. `repartitionByRange` is the practical choice when you want output files that are both evenly sized *and* clustered by a column, which makes downstream min/max statistics on Parquet row groups tight and predicate pushdown effective. Hash partitioning by the same column would balance nothing and would scatter adjacent values across every file. ## Custom partitioners On RDDs you can still subclass `Partitioner` — implement `numPartitions`, `getPartition`, and `equals`/`hashCode`. `equals` matters: Spark uses partitioner equality to decide whether two RDDs are already co-partitioned and can be joined *without* a shuffle. A hand-written partitioner is the standard answer to a known hot key (route it to a dedicated partition, or spread it across several) or to a domain layout (all of one tenant's data together). It is a genuine differentiator answer rather than everyday knowledge, and it does not exist in the DataFrame API. ## Picking between them Hash when you need co-location by key and the keys are reasonably uniform — that is most joins and aggregations, and it is what Spark does by default. Range when you need ordering, or when you need evenly-sized partitions from a lumpy but not single-key-dominated distribution, and you can absorb the sampling pass and the non-determinism.