When producing events to a partitioned topic, what determines which partition a given message is written to, and how should you pick a partition key so that all events belonging to the same business entity (say, a shopping cart or an IoT device) stay in order relative to each other?

## How the partition is chosen Every partitioned log needs a deterministic rule for deciding which partition a given message lands on, because that decision is what creates (or breaks) ordering guarantees for related events. The mechanism, using Kafka as the concrete reference, is: a producer can attach an optional **key** (an arbitrary byte string) to each message. - **If a key is present**, the default partitioner hashes it with a consistent hash function and computes `partition = hash(key)` modulo the current partition count; every message with that exact key value is therefore routed to the same partition, deterministically, for as long as the partition count doesn't change. - **If no key is supplied**, the producer falls back to round-robin or sticky-batch distribution across partitions purely for load balancing, with no ordering relationship implied between messages at all. ## Why the key exists This exists because a partition's ordering guarantee is only useful if you can control which events end up co-located. Partitioning was introduced for parallelism — spreading write and read load across many independent logs — but plenty of real workloads have events that are causally related and must be seen in order by any consumer: - a shopping cart's "item added" must be seen before its "checkout completed"; - a device's "temperature reading" events need to reach a monitoring consumer in temporal sequence to detect a trend correctly; - an account's "debit" and "credit" events must be applied in the order they occurred or the balance is wrong. The **partition key** is the tool that lets you say, in effect, "everything about entity X goes on the same conveyor belt," converting the topic-wide free-for-all into per-entity ordered streams, while still letting different entities' streams run in parallel on different partitions. ## Ordering correctness against load distribution The core trade-off in key selection is between ordering correctness and load distribution. You want a key with high enough **cardinality** (many distinct values) that traffic spreads roughly evenly across all partitions — using something like "region" with only 3 values on a 24-partition topic wastes 21 partitions and creates 3 hot partitions. But the key must also be the right **causal grouping unit** — the identifier of whatever entity actually needs order preserved. These two requirements can conflict: an extremely hot entity (a viral post, a whale trading account) can still overload a single partition no matter how well-distributed the overall key space is, because all of that one entity's traffic is deliberately concentrated by design. That's an intentional and acceptable trade in most systems, because giving up ordering for a hot entity to relieve its partition would break the very guarantee you needed the key for. ## Key stability against flexibility A second trade-off is key stability versus flexibility. Because `hash(key) % numPartitions` determines placement, the key value itself must never change for the life of the entity you're ordering — if "orderId" is your key today but a refactor swaps it for "customerId" tomorrow, existing in-flight order histories that relied on the old key's colocated ordering are effectively broken for any consumer that assumed continuity. Similarly, choosing a composite or derived key (e.g., "orderId + eventType") defeats the purpose if it produces different hash buckets for events that should stay together — accidentally partitioning "orderId-created" separately from "orderId-paid" is a common and easy-to-miss bug. ## Failure modes Failure modes in production tend to be silent rather than loud. 1. **The most common**: someone partitions by a low-cardinality field (event type, tenant tier, or nothing at all) "to spread the load," and later a downstream service that assumes causal ordering for an entity starts processing stale or out-of-order state, corrupting derived data with no exception thrown — the pipeline looks completely healthy while producing wrong results. 2. **Another**: a hot key creates a single overloaded partition that lags far behind its siblings, and consumer-lag monitoring per-partition (not just topic-aggregate lag) is needed to even notice, since aggregate lag can look fine while one partition silently falls behind by hours. ## Where it shows up A concrete real-world pattern: - **change-data-capture tools like Debezium** key each database change event by the source row's primary key, guaranteeing that INSERT/UPDATE/DELETE events for one row stream through a single partition in commit order, so a downstream consumer rebuilding a materialized view never applies an UPDATE before its preceding INSERT; - similarly, **e-commerce order-processing pipelines** commonly key by orderId specifically so every lifecycle event for one order — created, paid, fulfilled, shipped — arrives at any consumer in exactly that sequence, even while thousands of other orders' events interleave across the topic's other partitions in parallel.