A teammate wants a single monotonic sequence number across all partitions of a topic to order events globally. Why is this impossible with offsets, and what are the alternatives?

## Why the request is impossible as stated Kafka offsets are **per-partition counters**. Partition 0 numbers its records 0,1,2…; partition 1 independently numbers its records 0,1,2…. There is **no shared counter** and **no cross-partition coordination** at append time. Therefore: - Offsets across partitions are **not comparable** — 'P0 offset 900' is neither before nor after 'P1 offset 12'. - There is **no topic-wide monotonic sequence**. This is a **design choice, not a limitation to fix**. A global gap-free sequence would force every producer to obtain the next number from a single coordinating authority before each append — a global serialization point. That reintroduces exactly the bottleneck partitioning exists to remove, capping write throughput at what one coordinator can do and adding a single point of contention/failure. Kafka instead guarantees **ordering and monotonic offsets only within a single partition**, which is what makes it horizontally scalable. ## What ordering Kafka does guarantee - **Within a partition:** total order; offsets strictly increasing; a consumer reads records in offset order. - **Across partitions:** none. Consumers may interleave partitions arbitrarily. ## Alternatives (with tradeoffs) ### 1. Single partition Put everything in one partition. You get a perfect total order (offset == global sequence). Cost: **no parallelism** — one consumer in a group can read it, throughput is capped by one broker/partition, and you lose Kafka's main scaling lever. Acceptable only for low-volume streams that genuinely need total order. ### 2. Partition by key (per-entity ordering) Give records a **key** (e.g. accountId, orderId). The default partitioner hashes the key so all records for a key land in the **same** partition and are totally ordered **relative to each other**. Different keys may interleave, but you usually only need ordering *per entity*, not globally. This is the idiomatic answer and preserves parallelism (up to the partition count). ### 3. Application-level ordering value + downstream reorder Embed a monotonic value in the payload and reorder after consumption: - **Event-time timestamp** (and use Kafka Streams / Flink event-time windows with watermarks to handle out-of-order arrival). - **Producer-side sequence number** per logical stream. - **External global ID**: a database sequence, or a **Snowflake-style ID** (time + node + counter) that is globally roughly-monotonic without a central bottleneck on every write. Consumers buffer across partitions and emit in sorted order within a bounded window. Cost: **latency** (you must wait for laggards), **memory** (buffering), and **complexity** (handling late/duplicate events). ## Decision guidance (principal lens) - Ask **what ordering is actually required**. 'Global order' is usually over-specified; per-entity order suffices -> key-partition. - If true total order is non-negotiable and volume is small -> single partition. - If you need global order *and* scale -> accept eventual/approximate order via timestamps + downstream windowing, and design idempotent/commutative processing so exact global order matters less. - Beware false 'solutions': a global counter service, distributed locks, or `acks=all` do **not** create cross-partition order. ## Edge cases - Even within a partition, **retries with `max.in.flight.requests > 1` and no idempotence** can reorder; enable the idempotent producer (`enable.idempotence=true`) to preserve per-partition order under retries. - Compaction and transaction markers create offset gaps, so even a single-partition 'sequence' isn't perfectly contiguous.