skip to content

How does Kafka achieve fan-out to multiple independent consumers, and how does that differ from a fanout exchange in RabbitMQ or SNS-to-SQS?

level: middleimportance: must knowfreq 70%

answer

  1. group = own offsets = full copy
  2. same group = split partitions (parallelism)
  3. fanout exchange = physical copies
  4. new subscriber: Kafka replays, SNS doesn't
  5. partitions cap intra-group parallelism

basics

~10 s

In Kafka each consumer group has its own offsets, so many groups read the same topic without copying data. RabbitMQ/SNS fan out by duplicating each message into multiple queues at publish time.

solid answer

~50 s

Kafka fan-out is read-side: a topic is read by multiple **consumer groups**, each tracking its own offsets in `__consumer_offsets`. Every group sees every record independently — no data is duplicated; they just read the same log at different positions. *Within* a group, partitions are distributed across members for parallel, load-balanced consumption (one partition → at most one consumer in the group at a time). RabbitMQ does write-side fan-out: a **fanout exchange** copies each published message into every bound queue, so N subscribers means N physical copies and N independent ack/delete lifecycles. SNS→SQS is similar: SNS publishes a copy to each subscribed SQS queue. The trade-offs: Kafka fan-out is cheap (one stored copy, add groups freely, replayable) but a new subscriber must read from an offset that still exists in retention; RabbitMQ/SNS fan-out gives each subscriber an independent queue with per-message redelivery and DLQs, but storage/coupling grows per subscriber and there's no historical replay.

go deeper

for a junior

Know that different consumer groups each see all messages; same group splits the work.

for a middle

Explain offsets-per-group, partition assignment, and the write-side vs read-side fan-out contrast.

for a senior

Reason about rebalance cost, partition-count planning, and when DLQ-style per-message semantics push you toward a queue.

for a principal

Weigh a Kafka event backbone (one source of truth, replay) against per-subscriber queues for delivery isolation across an org's services.

## Definitions - **Consumer group:** a named set of consumer instances that cooperate to read a topic. Kafka assigns each partition to exactly one consumer *within* a group at a time (the **group coordinator** + a rebalance protocol manage assignment). Offsets are committed *per group*, so the group's progress is a single shared cursor across its partitions. - **Fanout exchange (RabbitMQ):** an exchange type that ignores routing keys and copies every message to *all* bound queues. - **SNS→SQS fan-out:** an SNS topic with multiple SQS subscriptions; each publish delivers a separate copy to each queue. ## Kafka fan-out (read-side) One topic, one physical copy of the data (replicated for durability, but logically one log per partition). If team A and team B each create their *own* consumer group on topic `orders`, both read every record independently — A's offset and B's offset are unrelated. Adding a third consumer group costs almost nothing: no extra storage, no producer change. This is why Kafka is described as 'publish once, subscribe many,' and why it's a good backbone for event-driven systems and stream processing. **Parallelism vs fan-out are orthogonal:** scaling *throughput* within one logical subscriber = add consumers to the *same* group (up to the partition count). Getting an *independent full copy* of the stream = use a *different* group. A common mistake is putting two services in the same group expecting both to get all messages — instead they split the partitions and each gets a subset. ## RabbitMQ / SNS fan-out (write-side) The broker materializes a copy of each message into each subscriber's queue at publish time. Each queue then has its own lifecycle: independent acks, redelivery, dead-letter queues, TTLs. This is great when subscribers need strong per-message delivery guarantees and isolation, and when you never need to replay history. But storage and broker work scale with the number of subscribers, and a new subscriber only sees messages published *after* it subscribed — there's no rewind. ## Trade-offs summary | Aspect | Kafka (groups) | RabbitMQ/SNS (fanout) | |---|---|---| | Copies of data | One log, many cursors | One per subscriber queue | | New subscriber sees history | Yes, within retention | No (only future messages) | | Per-message redelivery/DLQ | Coarser (offset-based, app-managed) | Rich, built-in | | Cost of adding subscribers | ~free | Grows per subscriber | ## Edge cases - In Kafka, max useful parallelism within a group = number of partitions; extra consumers sit idle. - A rebalance (member join/leave, or static-membership timeout) pauses consumption briefly; use cooperative-sticky assignor and `group.instance.id` to reduce churn. - If a brand-new Kafka group sets `auto.offset.reset=latest`, it will *not* see historical records — a frequent 'why did my new consumer miss data?' bug.

  • Two services join the same Kafka consumer group expecting each to get all messages. What actually happens?
    They share the partitions — each service gets only a subset of messages. To each get the full stream, they need separate consumer groups.
  • What limits how many consumers in one group can do useful work in parallel?
    The partition count. A partition is assigned to at most one consumer per group, so consumers beyond the partition count sit idle.

saying these in an interview costs you the question

  • Saying multiple consumers in the SAME group each receive every message (they split partitions).
  • Claiming a new RabbitMQ/SNS subscriber can replay past messages.
  • Confusing within-group parallelism with cross-group fan-out.

context

resolving links…