What guarantees does Kafka provide about offset ordering within a partition, and how do segments preserve it?

## The core guarantee An **offset** is the position of a record within its partition. Kafka assigns offsets on the leader at append time, strictly in the order records are appended. The guarantee is **per-partition total order**: if record A is appended before record B in the same partition, then offset(A) < offset(B), and that ordering is permanent because records are immutable. There is **no ordering guarantee across partitions** of the same topic — that's the price of partition-level parallelism. To get ordering for related records, you route them to the same partition (usually via a shared key, since partition = hash(key) % numPartitions by default). ## How segments keep it continuous The partition log is split into segments, but the offset space is one continuous sequence across them. When the active segment rolls at last offset M, the new segment's **base offset** is M+1 (the offset of its first record). So reading segment files in base-offset order reproduces the exact append order with no gaps at segment boundaries. This is why locating a record is a two-step floor lookup: find the segment whose base offset is the largest <= target, then use that segment's .index. ## Increasing != gapless A subtle senior point: offsets are **monotonically increasing** but **not guaranteed contiguous**. Holes can appear because: - **Log compaction** deletes superseded records by key, leaving gaps where they used to be. - **Transactions** write control records (commit/abort markers) that consume offsets but are invisible to consumers. - **Aborted transactions** leave records whose offsets a read-committed consumer skips. So a consumer may see offsets 5, 7, 8, 12 — increasing, with gaps. Code must never assume offset+1 is the next visible record. ## Tail markers - **Log End Offset (LEO):** the offset that will be assigned to the next record — i.e., one past the last record. - **High Water Mark (HWM):** the highest offset that has been replicated to all in-sync replicas and is therefore visible to consumers. Consumers cannot read past the HWM. The active segment holds records between HWM and LEO that are written but not yet fully committed. ## Why it matters Monotonicity is what makes committed offsets meaningful: a consumer that commits offset 100 can resume at 100 and know it will see every record after it, in order, exactly once relative to position. It also underpins replication (followers fetch by offset) and exactly-once semantics. ## Edge cases - After unclean leader election, a new leader may truncate uncommitted records, so the LEO can move backward on a replica — but committed offsets (<= HWM) never reorder. - Producer retries with idempotence enabled prevent duplicate offsets for the same record; without idempotence, duplicates get distinct increasing offsets.