In Kafka Streams exactly-once, how did zombie fencing evolve from per-partition transactional.ids to a single producer per instance, and why?

## Background: what fencing must guarantee in Streams Kafka Streams does consume-process-produce. For exactly-once, after a rebalance the **new owner** of a partition must be able to produce/commit while the **old owner (zombie)** must be fenced. The question is how to scope the `transactional.id` and producers so that fencing is both **correct** and **scalable**. ## EOS v1 (`exactly_once`): per-partition transactional.id - Streams assigned a **distinct `transactional.id` per input partition/task**, e.g. derived from `applicationId` + the task id. - Each such id mapped to its own **producer instance**, its own coordinator interaction, and its own transaction. - **Fencing was per-task:** after a rebalance, the new task's `InitProducerId` bumped the epoch for *that* id, fencing the old task's producer. - **Why it didn't scale:** the number of producers (and TCP connections, coordinator state entries, memory) grew **linearly with the number of partitions**. A topology with many partitions meant a flood of producers, slow restoration, and longer, costlier rebalances. ## EOS v2 (`exactly_once_v2`, KIP-447): one producer per thread - A **single producer per stream thread/instance** now drives a transaction that spans **all** partitions that thread owns — far fewer producers and connections. - Fencing no longer relies on a per-partition id epoch bump alone; it leverages the **consumer group generation**. When the producer commits consumer offsets via **`sendOffsetsToTransaction(offsets, consumerGroupMetadata)`**, it passes the consumer's **group metadata (member id + generation)**. - The **group coordinator and transaction coordinator cooperate**: a commit from a producer whose consumer is from an **older generation** (a zombie that missed a rebalance) is **rejected**. So a stale instance can't commit even though it might still hold a valid producer epoch for its own id. - Net effect: correctness is preserved (single writer per partition), but the producer count is decoupled from partition count. ## Why this matters architecturally - **Scalability:** EOS v2 makes exactly-once viable for large topologies; rebalances are cheaper and resource use is bounded by thread count, not partition count. - **Two-coordinator cooperation:** the design shows fencing as a *composition* — producer **epoch** (transaction coordinator) plus consumer **generation** (group coordinator) — rather than relying on epoch alone. - **Migration:** moving from v1 to v2 requires brokers ≥ 2.5 and is a one-way upgrade; `exactly_once` and `exactly_once_beta` are deprecated aliases superseded by `exactly_once_v2`. ## Edge cases / gotchas - A long GC pause can make an instance miss a rebalance; v2 fences it at **commit time** via generation mismatch, even if its producer epoch looks current to the transaction coordinator. - Because one transaction spans many partitions, an abort rolls back more work — tuning `transaction.timeout.ms` and `commit.interval.ms` matters for throughput vs. blast radius. - The transactional.id is now per **thread/instance**, so the deterministic id->coordinator mapping still holds, just at coarser granularity. ## Takeaway Fencing went from 'one id/producer per partition with per-task epoch fencing' to 'one producer per instance with generation-aware fencing,' trading a scalability bottleneck for a small increase in abort blast radius and a dependency on consumer-group/transaction-coordinator cooperation.