skip to content

Define base offset, log-start-offset, and log-end-offset (LEO) for a Kafka partition. How do they move over time?

level: middleimportance: should knowfreq 55%

answer

  1. base offset = first offset of a segment (in filename)
  2. log-start = earliest retained, moves right on delete
  3. LEO = next offset = last + 1
  4. leader LEO grows on append; followers lag
  5. HW = min(ISR LEOs) <= LEO

basics

~20 s

Log-start-offset is the offset of the earliest record still retained. LEO (log-end-offset) is the offset that will be assigned to the next record — one past the last written record. Base offset is the first offset of a particular log segment file.

solid answer

~50 s

A partition log is a series of segment files. Each segment's **base offset** is the offset of its first record and is encoded in its filename (e.g. 00000000000000012345.log). The **log-start-offset** is the smallest offset still available to read; it advances forward as retention/compaction deletes old segments or as DeleteRecords is called. The **log-end-offset (LEO)** is the offset that the *next* appended record will receive — i.e. one greater than the last written record's offset (it equals the count of records ever appended in a gapless log, minus deletions don't affect it). The leader's LEO grows on every append. Followers maintain their own LEO as they fetch; the leader tracks each follower's LEO to compute the high watermark. So: log-start-offset advances from the left (deletion), LEO advances on the right (appends), and base offsets are fixed per segment.

go deeper

for a junior

Know roughly that there's a 'start' (earliest available) and an 'end' (next offset) of the log.

for a middle

Define base offset, log-start-offset, and LEO precisely, including LEO = last + 1 and how each moves.

for a senior

Tie LEO to per-replica replication and the high watermark; explain OFFSET_OUT_OF_RANGE and deleteRecords.

for a principal

Use the segment/base-offset/index model to explain O(log n) offset and timestamp lookups and retention mechanics.

## The log is segmented A partition isn't one giant file. Kafka stores it as a sequence of **segment** files. When the active segment reaches `segment.bytes` or `segment.ms`, it is rolled and a new active segment starts. Only the newest (active) segment is being appended to. ## Base offset The **base offset** of a segment is the offset of the **first record in that segment**. Kafka encodes it as a zero-padded 20-digit number in the file names: `00000000000000012345.log`, `.index`, `.timeindex`. So a segment whose base offset is 12345 holds records starting at offset 12345. Base offsets are fixed: a segment never renumbers. The index files map offsets/timestamps to byte positions *relative to the base offset*, which is why lookups are fast. ## Log-start-offset (a.k.a. log start) The **log-start-offset** is the **smallest offset still retained** and readable in the partition. It begins at 0 for a fresh partition and **moves forward** when: - **Retention** (`retention.ms` / `retention.bytes`) deletes whole old segments. - **Log compaction** plus cleanup removes old data. - An admin/client calls **`deleteRecords`** (the `kafka-delete-records.sh` CLI / `Admin.deleteRecords`) to trim a prefix. Consumers that ask for an offset **below** the log-start-offset get an `OFFSET_OUT_OF_RANGE` error and fall back to their `auto.offset.reset` policy (`earliest` -> jump to log-start-offset, `latest` -> jump to LEO). ## Log-end-offset (LEO) The **log-end-offset (LEO)** is the offset that will be assigned to the **next** record appended — equivalently, *one past the last written record*. If the last record is at offset 99, the LEO is 100. On every successful append to the leader, the leader's LEO advances. Important: each **replica** has its own LEO. A follower's LEO is how far it has replicated. The **leader tracks all in-sync followers' LEOs**; the minimum of the leader and ISR followers' LEOs determines the **high watermark** (the boundary of committed, consumer-visible data). So LEO is the 'physical' end of the log, while the high watermark is the 'safe to read' end. ## How they move — summary - **base offset:** fixed per segment; set when the segment is created. - **log-start-offset:** advances from the left as old data is deleted/compacted/trimmed; never goes backward. - **LEO:** advances on the right with every append; the leader's LEO is the true end; followers lag behind. ## Edge cases - A brand-new partition: log-start-offset = 0 and LEO = 0 (no records, next append is offset 0). - After deleting everything via retention, log-start-offset can equal LEO (empty but offsets not reset). - LEO is **not** the last record's offset — it's last + 1. Off-by-one here is a classic interview slip. - Followers' LEOs ≤ leader's LEO; the high watermark ≤ leader's LEO. You can never read past the high watermark even though LEO is higher.

  • If the last record in a partition is at offset 99, what is the LEO?
    100. LEO is the offset the next append will receive, i.e. last written offset + 1, not 99.
  • What happens if a consumer requests an offset below the log-start-offset?
    It gets OFFSET_OUT_OF_RANGE and applies auto.offset.reset: 'earliest' seeks to the current log-start-offset, 'latest' seeks to the LEO, 'none' throws an exception.

saying these in an interview costs you the question

  • Saying LEO equals the last record's offset (it's last + 1).
  • Claiming log-start-offset can move backward.
  • Confusing LEO (physical end) with the high watermark (committed/consumer-visible end).
  • Thinking base offset changes as data is deleted.

context

resolving links…