Offset Addressing Model

## What a partition is A Kafka **topic** is split into one or more **partitions**. Each partition is an ordered, append-only sequence of records — conceptually a log file you can only add to at the end. Producers append records; the data is immutable once written. ## What an offset is An **offset** is the position of a record within its partition. The very first record in a partition has offset `0`; the next has `1`, then `2`, and so on. The offset is a 64-bit integer (`long`). It is: - **Monotonically increasing** — each new append gets a strictly larger offset than the previous one. - **Assigned by the partition leader** at the moment the record is appended to the log. The producer does not choose it. - **Immutable** — once a record sits at offset 42, it is offset 42 forever; offsets are never reused or renumbered. - **Dense in a normal log** — consecutive, no gaps. (Exception: **log compaction** removes superseded records, leaving offset gaps; transactional topics also consume offsets for commit/abort control markers, so you can see small gaps even in non-compacted topics.) ## Why offsets are partition-local Each partition maintains its **own** offset counter, starting from 0 independently. So: - Offset `5` in partition `0` and offset `5` in partition `1` are **completely unrelated records**. - You **cannot compare** offsets across partitions: 'offset 1000 in P0' is not 'later' or 'earlier' than 'offset 30 in P1'. They live in different number lines. - There is **no global offset** for a topic. Kafka deliberately avoids a global sequence number because that would require cross-partition coordination and destroy horizontal scalability. ## Consequences - **Ordering guarantee:** Kafka guarantees ordering only *within* a partition. Records that must be ordered relative to each other must land in the same partition (typically by sharing a key). - **Consumer progress** is tracked per `(topic, partition)`. A consumer's committed offset for partition 0 says nothing about partition 1. - **Parallelism:** because each partition is an independent log with its own offsets, partitions can be consumed in parallel by different consumers in a group. ## Edge cases - A brand-new partition has no records; its first append will be offset 0. - After retention deletes old segments, the lowest available offset (the **log-start-offset**) moves forward, but the offsets of remaining records do not change. - Compaction and transaction markers can make offsets non-contiguous, so consumers must never assume `nextOffset == lastOffset + 1`.