Walk through how a closed log segment gets offloaded to remote storage. What does the broker upload, and when can it delete the local copy?
answer
- closed + below high watermark + leader only
- STARTED then copyLogSegmentData then FINISHED
- upload = .log + offset/time/txn index + producer snapshot + leader-epoch
- upload != delete; local.retention governs local deletion
- segment can live in both tiers briefly
basics
~20 sWhen a segment rolls (closes), the leader broker's RemoteLogManager uploads it plus its indexes to remote storage and records metadata. Only after the upload is confirmed (COPY_SEGMENT_FINISHED) can the local copy be deleted, subject to local retention.
solid answer
~40 sOffload is driven by the **RemoteLogManager (RLM)** on the partition **leader**. A background RLMTask periodically scans for **closed** (rolled) segments below the high watermark that aren't yet remote. For each, it writes a RemoteLogSegmentMetadata in state **COPY_SEGMENT_STARTED** via the RLMM, then calls **RSM.copyLogSegmentData**, uploading the **.log** segment plus its **offset index, time index, transaction index, producer-snapshot, and leader-epoch checkpoint**. On success it writes **COPY_SEGMENT_FINISHED**, making the segment readable from remote. Crucially, only segments whose data is fully replicated (below the high watermark) and committed are eligible. The local file is *not* deleted at upload time — local deletion is governed by **local.retention.ms/bytes**, so recent offloaded data still serves fast local reads until local retention expires. This separation means a segment can briefly exist in both tiers.
go deeper
Know that closed segments are copied to remote, and the local copy is deleted later, not immediately.
Describe the STARTED/FINISHED upload sequence, what files are uploaded, and the leader-only + below-high-watermark conditions.
Explain crash safety via the two-phase state, leadership-change reconciliation, and the independence of upload vs. local deletion.
Reason about durability ordering (FINISHED only after durable upload), the both-tiers window, and consistency under leader failover and object-store eventual consistency.
## Preconditions for offloading A segment is eligible to be uploaded to the remote tier only when: - It is **closed/rolled** — Kafka has stopped appending to it and started a new active segment (controlled by segment.bytes / segment.ms). The active segment is never offloaded. - Its data is **fully durable**: the segment's offsets are at or below the partition **high watermark** (replicated to the ISR). Offloading uncommitted data could expose records that get truncated. - Only the **leader** of the partition offloads; followers do not upload. ## The upload sequence The per-broker **RemoteLogManager (RLM)** runs an **RLMTask** per leader partition: 1. Identify the next eligible closed segment not yet in the remote tier (using RLMM to know what's already there). 2. Create a **RemoteLogSegmentId** and write a **RemoteLogSegmentMetadata** record with state **COPY_SEGMENT_STARTED** via the RLMM. At this point the segment is *not yet* considered available remotely. 3. Call **RemoteStorageManager.copyLogSegmentData(metadata, segmentData)**. The uploaded bundle includes: - the **.log** segment file (the records), - the **offset index** (.index) — maps relative offsets to byte positions, - the **time index** (.timeindex) — maps timestamps to offsets, - the **transaction index** (.txnindex) — aborted-transaction markers for read_committed, - the **producer-snapshot** — for idempotent/transactional producer state, - the **leader-epoch checkpoint** — epoch→offset lineage so reads/recovery respect epoch boundaries. 4. On confirmed upload, update the metadata to **COPY_SEGMENT_FINISHED**. Now reads can be served from remote. The two-phase state (STARTED → FINISHED) makes this **crash-safe**: if the broker dies between steps 2 and 4, the segment stays in STARTED, is treated as not-yet-remote, and is retried — no torn/partial segment is ever exposed. ## When the local copy is deleted Uploading does **not** immediately delete the local file. Local deletion is independent and governed by **local retention** (local.retention.ms / local.retention.bytes — the operational sizing of which belongs to a sibling topic). A segment that has been offloaded (FINISHED) **and** has exceeded local retention is removed from local disk. Until then, the segment exists in **both** tiers and reads are served locally (fast). Total/remote retention (retention.ms/bytes) governs when the **remote** copy is finally deleted (DELETE_SEGMENT_STARTED → RSM.deleteLogSegmentData → DELETE_SEGMENT_FINISHED). ## Edge cases - **Leadership change mid-offload**: the new leader's RLM reconciles via RLMM; a STARTED-but-not-FINISHED segment is re-uploaded. - **Index integrity**: indexes must be uploaded with the segment, otherwise remote reads can't translate offsets/timestamps to byte positions. - **Ordering**: segments are generally offloaded in offset order; the remote tier presents a contiguous history below the local tier.
- Why isn't the local segment deleted immediately after a successful upload?To keep recent data fast to read locally. Local deletion is governed separately by local.retention.ms/bytes, so an offloaded segment stays local (served fast) until local retention expires.
- What gets uploaded besides the .log records file?The offset index, time index, transaction index, producer-snapshot, and leader-epoch checkpoint — all needed so remote reads and recovery can translate offsets/timestamps and respect epoch lineage.
saying these in an interview costs you the question
- Saying the active segment can be offloaded — only closed/rolled segments are.
- Saying followers offload — only the leader's RemoteLogManager does.
- Claiming the local copy is deleted the instant upload finishes — local.retention controls that.
- Forgetting that index/snapshot files travel with the segment.
- Offloading data above the high watermark (uncommitted) — not allowed.