How does a read_committed consumer actually avoid delivering records from aborted transactions?

## The core fact: aborted records are not deleted When a transaction aborts, Kafka does **not** go back and erase the records it already wrote. They remain in the log at their original offsets. Hiding them from read_committed consumers is therefore a *read-time filtering* problem, solved by metadata the broker maintains and ships to the consumer. ## What the broker stores: the transaction index (.txnindex) Each log segment has a companion **transaction index** file (`.txnindex`). When a transaction aborts, the broker appends an entry capturing: - the **producerId** (PID) of the aborting producer, - the **first offset** of that producer's data in this transaction, - the offset of the abort marker / last relevant offset, - and the LSO at that point. This lets the broker, for any fetched offset range, quickly enumerate which (producerId, offset-range) spans were aborted. ## What the broker sends on a Fetch For a `read_committed` fetch, the `FetchResponse` for each partition includes: 1. The record batches, but only **up to the LSO**. 2. An **`abortedTransactions`** list: entries of `(producerId, firstOffset)` for transactions that aborted within the returned offset range. 3. The current LSO (so the consumer can compute lag and know the read ceiling). ## What the consumer does (Fetcher filtering) The consumer's fetch path (in `CompletedFetch` / the fetch collector) reconstructs which records to deliver: - It maintains a set of **currently-aborted producerIds**, seeded from the `abortedTransactions` list, advancing through the data in offset order. - Each record **batch** header carries the `producerId`, plus flags: `isTransactional` and `isControl`. - **Control batches** (commit and abort markers Kafka writes to mark transaction boundaries) are never delivered to the application; the consumer consumes them only to know when a producer's aborted span ends. - For each transactional data batch, if its producerId is in the aborted set for that offset region, the whole batch is **skipped**. Otherwise the records are delivered. - Non-transactional batches are delivered unconditionally. ## Why per-batch, per-producerId Filtering granularity is the **batch** (record set), keyed by **producerId**, because a transaction's writes from one producer to one partition form contiguous batches tagged with that PID. Multiple producers can interleave on one partition, so the producerId is what disambiguates whose records to drop. ## Lifecycle and cleanup Aborted records and their control markers persist in the log until normal **retention** (time/size) or **log compaction** eventually removes them. The `.txnindex` is maintained alongside segment files and is rebuilt/validated during recovery. The cost of read_committed is thus mostly: extra metadata in fetches, a small consumer-side filtering pass, and storage of data that will never be delivered until retention reclaims it. ## Edge cases - **Cross-segment transactions**: aborted entries are tracked per segment, so a transaction spanning segments contributes entries to each. - **Consumer seeks**: when a read_committed consumer seeks into the middle of a range, the broker still ships the aborted-transaction list overlapping the new fetch position, so filtering remains correct after a seek. - **No double delivery of markers**: control records are filtered universally, independent of isolation level.