skip to content

Walk through exactly how Kafka resolves a fetch request for a specific offset using the sparse .index file.

level: seniorimportance: should knowfreq 35%

answer

  1. floorEntry on base offset -> segment
  2. binary search .index for offset <= target
  3. seek byte position, scan batches forward
  4. batch header: baseOffset + lastOffsetDelta
  5. scan bounded by index.interval.bytes

basics

~20 s

Kafka first picks the right segment (the one whose base offset is the largest <= target). It binary-searches that segment's .index for the entry with the largest offset <= target, seeks to that byte position in the .log, then scans record batches forward until it reaches the requested offset.

solid answer

~50 s

Resolution is two-level. (1) Segment selection: the partition's segments are kept in a skip-list keyed by base offset; Kafka finds the segment whose base offset is the floor of the target offset. (2) Within that segment: the .index is a sorted array of (relativeOffset, position) entries, memory-mapped, so Kafka binary-searches it for the entry with the largest relativeOffset <= (target - baseOffset). That gives a starting byte position in the .log; Kafka seeks there and scans forward batch-by-batch (reading the record batch headers, which carry baseOffset and lastOffsetDelta) until it locates the batch/record containing the target. Because the index is sparse (~one entry per index.interval.bytes), the scan covers at most one interval of log. A fetch typically returns whole record batches starting at or before the requested offset; the consumer skips earlier offsets. Recent Kafka also warms/caches indexes and binary-searches a hot vs cold region to keep page-cache friendly.

go deeper

for a junior

Know that a lookup uses the index to jump near the offset, then scans.

for a middle

Describe binary search + floor entry + short forward scan.

for a senior

Cover segment selection, batch-header scanning, mmap, and why earlier offsets arrive.

for a principal

Discuss recovery/rebuild, page-cache behavior, and fetch-path implications at scale.

## Step 0: which segment? A partition is many segments. Kafka holds them in a concurrent skip-list map keyed by **base offset**. To find offset N, it does a `floorEntry(N)` — the segment whose base offset is the greatest value <= N. That's the segment that can contain N. ## Step 1: binary search the .index The segment's `.index` is a sorted array of fixed-size entries: each is **(4-byte relative offset, 4-byte physical position)**. Because entries are fixed-width and sorted ascending, Kafka memory-maps the file and runs a **binary search** for `target - baseOffset`. The search returns the entry with the **largest relative offset <= the target's relative offset** (a floor lookup). If the target is below the first entry, it starts at position 0. ## Step 2: seek and scan the .log The matched entry gives a **byte position** in the `.log`. Kafka `seek()`s there and reads **record batches** sequentially. Each batch header carries `baseOffset` and `lastOffsetDelta`, so Kafka can tell whether the target offset falls inside a batch without decompressing it. It advances batch by batch until it finds the batch containing N. Thanks to sparseness this forward scan spans at most ~`index.interval.bytes`. ## Step 3: return Kafka fetches return entire record batches beginning at or before the requested offset (it cannot split a compressed batch). The **consumer** then discards records before N client-side. This is why a fetch may deliver a few records the consumer already has. ## Edge cases & details - **Sparse miss is normal:** the exact offset is almost never an index entry; the floor + scan is the common path. - **Index can be corrupt/empty after crash:** on startup Kafka may rebuild indexes from the `.log` (sanity-check/recovery); the index is a derived, regenerable structure. - **Memory mapping:** indexes are `MappedByteBuffer`s, so the OS keeps hot pages resident; binary search touches O(log n) pages. - **Out-of-range:** if N is below the log start offset (after retention/compaction) the broker returns an offset-out-of-range error, triggering the consumer's auto.offset.reset.

  • Why might a consumer receive records with offsets lower than the one it requested?
    Fetches return whole record batches; the index/seek lands at or before the target and batches can't be split (especially when compressed), so the broker sends the full batch and the consumer skips earlier offsets client-side.
  • What happens if the .index file is missing or corrupt when a broker starts?
    The index is a derived structure; Kafka rebuilds it by scanning the .log during log recovery/sanity checks, so no data is lost — only startup is slower.

saying these in an interview costs you the question

  • Claiming Kafka scans the whole segment linearly (binary search bounds it).
  • Saying the index points exactly at the requested offset (it's a floor; a forward scan finishes the job).
  • Saying the broker filters out earlier offsets in a batch (the consumer does that; batches aren't split).

context

resolving links…