Message Queue vs Event Log

## Where the mechanism differs The mechanism differs at the storage layer. - **A queue broker** holds a message until a consumer acknowledges it (`ack`), at which point the broker physically deletes it from its store; while a message is 'in flight' it's typically marked invisible for a visibility-timeout window, and if the ack never arrives it's redelivered. - **An event log**, by contrast, appends every message to an immutable, ordered partition and never deletes on read - deletion happens only when a separate, time- or size-based retention policy expires a segment. Consumers of a log don't tell the broker 'I'm done'; they simply remember an **offset** (a position number) marking how far they've read, and can move that number forward, backward, or not at all without changing what's stored. ## Why the two models diverge This design split exists because the two models solve different problems. - **Queues model discrete units of work**: a task that should be picked up by exactly one worker, processed, and then be off the books - the classic 'competing consumers' pattern for load distribution. - **Logs model facts that happened**: an immutable record of history that many independent parties might want to observe, now or later, much like a database's write-ahead log exposed externally for anyone to tail. ## The trade-offs The trade-offs follow directly. | | A queue | A log | |---|---|---| | **Storage** | keeps storage small (consumed messages vanish) | costs more to operate (retention window means disk usage for however many days/GB you configure, replicated for durability) | | **Consumer** | keeps the consumer simple (no offset bookkeeping, no replay logic needed) | pushes offset management onto every consumer | | **History** | but you permanently lose history - you cannot add a second independent consumer after the fact and have it see what already happened, and you cannot recover from a downstream bug by reprocessing | but in exchange it supports replay for bug fixes and backfills, and it supports adding brand-new consumer groups years later with zero coordination with the producer | ## Failure modes in production In production, the failure modes look different too. - **A queue's classic failure** is the poison-pill message that keeps failing and getting redelivered until a dead-letter queue (DLQ) catches it after N attempts - a clean, per-message isolation mechanism. - **A log's classic failure** is a consumer that commits its offset too early and crashes, silently skipping unprocessed messages (at-most-once), or commits too late and reprocesses on restart (at-least-once, requiring idempotent handling). A slow log consumer doesn't block other consumers - it just accumulates lag on its own bookmark - whereas a queue backlog is a single shared pile that every consumer of that queue competes over. ## One event, two outcomes A concrete scenario makes the contrast vivid: an order-service publishes 'OrderPlaced.' - **Using SQS**, only the fulfillment worker that dequeues a given message ever sees it; if six months later a billing team wants to react too, they need a new fan-out subscription set up going forward, and they get nothing from before that point since nothing historical survived. - **Using Kafka**, order-service publishes to a topic; a fraud-detection team spun up half a year later can create a brand-new consumer group and read the exact same historical events from the beginning of the retention window, entirely independent of the original fulfillment consumer and without any change to the producer.