Compare the ordering and retention guarantees of Kafka versus a traditional queue, and explain the trade-offs each model forces.

## Ordering **Kafka:** A partition is a totally ordered log. Kafka guarantees order **only within a single partition**. A topic with P partitions has P independent ordered streams; there is **no cross-partition ordering**. The standard technique is **keyed partitioning**: the producer hashes the record key to choose a partition (`partition = hash(key) % numPartitions` with the default partitioner), so all records with the same key (e.g., same `customerId`) land in the same partition and are processed in order. This means ordering is *per key*, and parallelism is bounded by partition count. Caveats: changing the partition count rehashes keys (breaks the same-key→same-partition invariant for future records); and to truly preserve producer order on retries you need `enable.idempotence=true` (and historically `max.in.flight.requests.per.connection<=5` with idempotence, or `=1` without) so a retried batch can't be reordered. **Traditional queue:** A single queue is FIFO in principle, but the moment you add **competing consumers** (multiple workers pulling from one queue for throughput), strict end-to-end order is lost because workers process at different speeds and redeliveries reshuffle. To get ordering you either use one consumer (no parallelism) or a partitioned scheme: **AWS SQS FIFO queues** preserve order *per MessageGroupId* (analogous to Kafka's per-key ordering) but cap throughput; **RabbitMQ** can use a consistent-hash exchange or single active consumer for per-key/serial ordering. So both worlds ultimately trade global order for parallelism — Kafka just makes the partition the explicit, durable unit. ## Retention **Kafka:** Retention is **decoupled from consumption**. Records persist for `retention.ms` (default 7 days) or until `retention.bytes` is exceeded, regardless of whether consumers have read them. Compacted topics (`cleanup.policy=compact`) instead keep the latest value per key indefinitely (good for changelog/state). Consequences: - **Replay** is possible — seek to an earlier offset or timestamp and reprocess. - **Data loss risk for slow consumers:** if a consumer lags beyond retention, the oldest unread records are deleted before it reads them. Monitor **consumer lag** (e.g., via `kafka-consumer-groups.sh --describe`, Burrow, or lag metrics). - **Storage scales with retention × throughput**, not backlog — you pay to keep history even if everyone's caught up. **Traditional queue:** Storage holds the **backlog of unacked messages**. Once acked, gone. Messages can have a **TTL**; expired or repeatedly-failed messages go to a **dead-letter queue (DLQ)**. There's no replay of acked history. SQS has a max retention (up to 14 days) for *undelivered* messages, but that's a backlog cap, not a replay log. So storage is naturally bounded by how far consumers fall behind, and you reason about backpressure and DLQs rather than retention windows. ## Trade-off matrix | | Kafka | Queue (RabbitMQ/SQS) | |---|---|---| | Ordering | Per partition (per key) | FIFO degrades with competing consumers; per-group via FIFO/hashing | | Retention | Time/size, replayable, consumption-independent | Until ack/TTL; backlog-bounded; no replay | | Risk | Slow consumer drops off the log | Backlog growth / DLQ overflow | | Parallelism unit | Partition (fixed-ish) | Consumers on a queue (elastic) | | Reprocessing | Seek + replay | Re-enqueue / DLQ redrive | ## When each wins - **Kafka:** event sourcing, stream processing, multi-consumer analytics, anything needing replay or ordered-per-key high throughput. - **Queue:** task/work dispatch with per-message acks, elastic worker pools, rich retry/DLQ semantics, and no need to keep or replay history.