How does a broker's message retention policy differ between 'delete after acknowledgment' queues and 'keep for a fixed time/size regardless of consumption' log retention, and what does each let you do that the other doesn't?

## The two retention models - **The consumption-based retention model.** A message's lifecycle is tied directly to acknowledgment: the broker holds it, delivers it to a consumer, and upon receiving a positive acknowledgment, removes it from storage — the message ceases to exist after that point. The queue's storage footprint at any moment reflects only the current unconsumed backlog. - **The log-based retention model.** Published messages are appended to a durable, ordered log and kept according to a policy configured independently of consumption — typically 'keep for N days' or 'keep up to N gigabytes per partition' — and consumption never triggers deletion. Instead, each consumer tracks its own read offset into the log; the broker doesn't know or care whether a given message has been consumed by anyone, only that it hasn't yet aged out of the retention window. ## Why each one exists Consumption-based retention exists because, for pure task-distribution workloads, once a task is done there is nothing more to do with the message — keeping it forever would waste storage for no benefit. Log-based retention exists for event streaming, where the same events may need to be read by consumers that don't yet exist, by consumers at very different speeds, or by the same consumer intentionally rewinding after fixing a bug in its processing logic. None of that is possible if the broker deletes a message the instant it's first read. ## The mirrored trade-off The consumption-based model's trade-off is the mirror of the log-based model's: | Retention model | What it buys | What it charges | |---|---|---| | Consumption-based | storage stays small and predictable | you permanently lose the ability to onboard a new consumer needing anything from before it existed, and you can't replay to recover from a downstream processing bug | | Log-based | you gain replay and multi-speed flexibility | you pay for it in disk (retaining every message for days multiplies storage requirements) and in shifted responsibility — every consumer must track and persist its own offset and handle corruption or resets | With a log you must also choose a retention window carefully: too short and a slow or down consumer falls out of the window and permanently misses data; too long and storage costs balloon for data nobody reads again. ## Failure modes - **Reprocess history after the fact.** A common failure with consumption-based queues is discovering, only after the fact, that you need to reprocess history — e.g., a consumer had a bug that silently corrupted data for a week, but since the queue deletes on acknowledgment, there's no way to replay those already-consumed messages; the only backstop is an audit log kept elsewhere, if one exists. - **Offset mismanagement.** A common failure with log-based retention is a consumer's stored offset being lost or reset to the beginning, after which it can reprocess the entire retention window's worth of messages, causing downstream duplicate-processing storms if the consumer isn't idempotent. - **Falling behind the window.** Another log-retention failure is a consumer falling behind its retention window entirely — if it's down longer than the retention period, the oldest messages it needed have already been deleted by the time it returns, and it silently has a permanent gap, with no error raised by the broker. ## Two workloads, two fits - **A payroll-processing queue** (consumption-based) is the right fit for 'pay this employee, once' — once paid and acknowledged, there's no value in keeping that task around, and doing so would just waste storage; if you need an audit trail, that's a job for a separate database table. - **By contrast, a clickstream/event-tracking pipeline** (log-based, e.g. 7-day retention) is the right fit because a new analytics team six months from now might want to build a fresh aggregation job and, thanks to log retention, can replay the last week of events to backfill and validate their new consumer against real historical data before going live — something a consumption-deleted queue could never offer.