Explain the per-region local + aggregate consumption pattern in an active-active deployment. How should a globally-aware consumer read all data, and what are the aggregate-cluster trade-offs?
answer
- local topic vs remote (prefixed) topic
- regex subscribe: (^|.*\.)orders$
- aggregate cluster = one unified namespace
- extra hop = more lag + SPOF
- no global total order; per-partition only
basics
~20 sEach region has local topics (written there) plus remote mirrored topics from other regions. A region-local consumer reads just local topics; a globally-aware consumer reads both local and remote (prefixed) topics — via a regex subscription or by consuming a separate aggregate cluster that holds all regions' data in one place.
solid answer
~50 sIn active-active, every cluster holds local topics (produced in-region) and remote topics (mirrored in, carrying a source-alias prefix). Two consumption modes exist. (1) **Local-only**: a latency-sensitive app reads only its region's local topic — fast, but sees only local writes. (2) **Aggregate/global**: an app that needs the full global stream must read local + all remote forms. You achieve this either by a **regex subscription** (e.g. `subscribe(Pattern.compile("(^|.*\\.)orders$"))`) so the consumer picks up `orders`, `A.orders`, `B.orders`, or by standing up a dedicated **aggregate cluster** that MM2 mirrors all regions into, so consumers see one unified namespace. The aggregate-cluster pattern centralizes global reads and simplifies consumer config, but adds another cluster to operate, an extra replication hop (more end-to-end lag), and a potential single point of failure for global consumers. Regex-on-each-cluster avoids the extra cluster but pushes prefix-awareness into every consumer.
go deeper
Know each region has its own local data plus mirrored copies of other regions, and a 'see everything' consumer must read both.
Contrast local-only vs aggregate consumption, implement a regex subscription that catches prefixed topics, and name the aggregate-cluster pattern.
Weigh regex-per-cluster vs dedicated aggregate cluster on latency, availability, and operational cost; account for duplicates and lack of global ordering.
Design the regional topology (per-region aggregates for HA), define consumer contracts org-wide, and set SLOs around the extra replication hop's lag.
## Why two topic kinds exist on each cluster Active-active means both regions accept writes and MM2 replicates each way. After MM2's `DefaultReplicationPolicy` prefixing, cluster A ends up with: - **Local topic** `orders` — everything produced in region A. - **Remote topic** `B.orders` — everything produced in region B, mirrored in. Cluster B is the mirror image: local `orders`, remote `A.orders`. ## Consumption mode 1 — local-only (per-region) A consumer that only cares about its own region (e.g., a regional fulfillment service) subscribes to just `orders` on its home cluster. **Pros**: lowest latency (no replication hop), no cross-region dependency. **Cons**: it is blind to writes in the other region. This is correct *only* when the domain is genuinely region-partitioned. ## Consumption mode 2 — aggregate / global A consumer that needs *all* events globally (analytics, fraud, a global materialized view) must read local **and** remote topics. Two implementations: ### (a) Regex subscription on each cluster The consumer subscribes by pattern so it captures the local and all prefixed remote variants: ``` consumer.subscribe(Pattern.compile("(^|.*\\.)orders$")); ``` This matches `orders`, `A.orders`, `B.orders`. The Kafka client periodically refreshes metadata, so newly created remote topics are picked up automatically. **Pros**: no extra cluster. **Cons**: every global consumer must encode the prefix grammar; pattern mistakes silently drop data; the consumer still only sees what *its* cluster has mirrored. ### (b) Dedicated aggregate cluster Stand up a separate cluster (often per-region or one central) that MM2 mirrors **all** source clusters into. Global consumers point only at the aggregate cluster and see a single, complete namespace. This is the classic **aggregate-cluster pattern** from large multi-DC Kafka deployments. **Pros**: clean separation of local (low-latency, in-region) vs. global (aggregate) reads; consumers don't need prefix logic if you also normalize names; one place to scale global read load. **Cons / trade-offs**: - **Extra hop = more lag**: data travels region → aggregate, adding end-to-end latency and another point where lag can build (watch MM2's `replication-latency-ms` and consumer lag on the aggregate). - **Operational cost**: another cluster + MM2 flows to run, monitor, secure. - **Availability**: if the aggregate cluster is single and it's down, *all* global consumers stall; HA usually means an aggregate per region (each aggregating all sources), doubling replication. - **Duplicates**: aggregate still contains both local and remote copies; if a record was dual-written or re-mirrored, the aggregate sees duplicates and consumers need idempotent processing / dedup keys. - **Ordering**: the aggregate interleaves regions; there is no global total order across partitions/regions — only per-partition order within each source topic is preserved. ## Choosing Use **local-only** for region-partitioned workloads, **regex** when you have few global consumers and want to avoid extra infrastructure, and the **aggregate cluster** when many consumers need the global stream, you want to isolate global read load, or you need a stable single namespace. Many shops run local clusters for serving traffic and a per-region aggregate for analytics.
- How does a regex-subscribing consumer discover a brand-new remote topic that MM2 just created?The Kafka consumer periodically refreshes topic metadata (metadata.max.age.ms) and re-evaluates the subscription pattern, so newly created matching topics (like a new 'C.orders') are auto-assigned without code changes — though there is a refresh-interval delay before they're picked up.
- Your aggregate cluster shows higher consumer lag than the source clusters. Where do you look?The aggregate adds a replication hop, so check MM2's replication-latency-ms and record-age-ms metrics for the flows into the aggregate, the aggregate brokers' load, and whether global consumers are simply slower than producers. The extra hop inherently raises end-to-end lag versus reading the source directly.
saying these in an interview costs you the question
- Assuming a single subscribe('orders') in one region sees global data — it only sees local writes
- Claiming an aggregate cluster gives a global total ordering of events across regions
- Forgetting the aggregate cluster adds latency and is a SPOF for global consumers if not made HA
- Ignoring that the aggregate still contains duplicates from dual-writes/re-mirroring and needs dedup