You run a 3-AZ Kafka cluster on AWS and your bill is dominated by cross-AZ data transfer from consumers. As a principal engineer, how do you design a fetch-from-follower rollout to actually capture the savings, and what can undermine them?

## Where the money goes In a multi-AZ AWS Kafka cluster there are three cross-AZ traffic sources: (a) **producer → leader** (often cross-AZ), (b) **leader → follower replication** (cross-AZ by design, for durability), and (c) **leader → consumer reads** (cross-AZ when the consumer isn't co-located with the leader). KIP-392 fetch-from-follower attacks **only (c)** — the consumer-read slice. For fan-out-heavy workloads (many consumer groups), (c) is frequently the largest line item, which is why FFF can be a big win — but it does **not** remove (a) or (b). ## Designing the rollout ### Step 1 — Honest rack labeling Set `broker.rack` on every broker to its true AZ (`us-east-1a/b/c`). This label is the locality key. A typo or stale value silently routes reads cross-AZ with no error — only the bill reveals it. Treat rack as config-managed and validated across the fleet. ### Step 2 — Placement must cover every consumer AZ The selector can only return a replica that **exists and is in-sync in the consumer's rack**. So replica **placement** must put a replica of each partition in each AZ that has consumers. Practically: use **rack-aware assignment** and pick a replication factor that covers your AZs (e.g., RF=3 across 3 AZs). RF=2 across 3 AZs leaves one AZ with no local replica for some partitions → those reads stay cross-AZ. Placement and selection are two halves of one design; either alone is insufficient. ### Step 3 — Enable the selector cluster-wide `replica.selector.class=RackAwareReplicaSelector`, applied via dynamic broker config (or rolling restart), validated on all brokers. Missing it on even some brokers leaves those leaders serving cross-AZ. ### Step 4 — Consumer client.rack Every consumer sets `client.rack` to its own AZ. In Kubernetes, derive it from the node's topology label (`topology.kubernetes.io/zone`) via the downward API so pods self-configure correctly regardless of where they're scheduled. ### Step 5 — Measure, don't assume Validate with **cross-AZ byte metrics / VPC flow logs / cost reports**, not just config. Confirm `preferredReadReplica` is being honored (consumer metrics) and watch the cross-AZ transfer line drop. ## What undermines the savings - **No in-rack replica** for some partitions → those reads remain cross-AZ. - **Rack misconfiguration** → silent cross-AZ reads, no error. - **Leader/ISR churn** → temporary fallback to the leader (cross-AZ) until consumers re-home; frequent rebalancing erodes savings. - **Hot partitions / skew** → if consumers cluster in one AZ but leaders/replicas don't, coverage gaps. - **Ignoring (a) and (b)** → produce and replication remain cross-AZ; FFF is not a total-egress solution. - **Latency-sensitive consumers** forced onto followers pay in freshness; sometimes correct answer is to keep them on the leader. ## Strategic framing FFF is one tool in cost-optimized multi-AZ design. Complementary moves: co-locating producers near leaders, accepting replication cost as the price of durability, and (for cross-*region* DR) using MirrorMaker 2 rather than stretching one cluster. FFF keeps leaders single-region while letting reads be AZ-local — exactly the 'leaders stay put, reads go local' pattern.