A claim-check pipeline has been running for six months and the object storage bucket backing it has grown to 40 TB, far more than the traffic volume would suggest. What's the likely root cause, and how would you diagnose and fix it?

## Why the bucket keeps growing Unbounded storage growth in a claim-check pipeline is almost always a symptom of the payload lifecycle never being closed out -- every object written for a claim check is a liability until something explicitly removes it, and if nothing ever does, the bucket grows monotonically for as long as the pipeline runs, regardless of how much traffic is currently flowing through it. This is a distinct failure mode from anything the broker itself would surface, because the broker has no visibility into the storage system at all; a broker dashboard showing healthy, low queue depth gives no signal whatsoever about whether the storage side is quietly accumulating. ## The usual root cause: cleanup designed for the happy path The most common root cause is that cleanup was designed around the happy path only: a consumer deletes the object from storage after it finishes processing successfully, and that's the only deletion path that exists anywhere in the system. Every payload becomes permanently orphaned when it is: - one whose processing failed; - one whose consumer crashed before reaching the delete call; - one whose message was redelivered and processed by a different consumer instance that also forgot to delete; - or one whose consumer was simply never written to delete at all. In a **fan-out topology**, where several independent consumers each need to read the same payload -- for example separate transcoding, thumbnailing, and moderation services all reading the same uploaded video -- this gets structurally worse, because no single consumer can correctly delete the object without knowing whether every other subscriber has finished, and naive implementations often just skip deletion altogether to avoid that coordination problem, guaranteeing every object lives forever by default. ## How to diagnose it Diagnosing this starts with characterizing the growth: sample a set of objects by age and compare their creation timestamps against processing logs or a database table that records when each corresponding message was successfully consumed. - If a large fraction of objects older than the pipeline's expected processing SLA are still present and their corresponding processing did complete, that confirms deletion isn't happening on the success path. - If a smaller but still significant fraction correlates with failed or dead-lettered messages, that points at error-path cleanup being the gap specifically. It's also worth checking whether the bucket has any lifecycle configuration at all -- surprisingly often, teams build the write and fetch sides of a claim-check pipeline carefully and never configure the storage system's own expiry rules, treating cleanup as purely an application-level concern. ## The robust fix The robust fix is to stop treating consumer-side deletion as the primary or only cleanup mechanism and instead configure a **lifecycle policy** directly on the storage bucket or container -- most object stores support rules like 'delete objects older than N days' or 'transition to cheaper cold storage after N days, then delete after M more' natively, entirely independent of application code. - **The backstop.** This guarantees an upper bound on storage growth regardless of whether any consumer's delete call ever runs, which matters because it removes an entire category of failure (crashed consumers, buggy delete logic, forgotten error paths) from the equation. - **The optimization on top.** Application-level deletion can still be layered on top as an optimization -- deleting immediately after successful processing reduces storage cost sooner than waiting for the lifecycle policy's expiry window -- but it should never be the only safety net. - **Fan-out topologies.** For fan-out topologies specifically, the retention window needs to be set generously enough that the slowest expected consumer has time to finish before expiry, or the pipeline needs an explicit completion-tracking mechanism (a counter or a set of consumer IDs that have acknowledged processing) before any deletion, whether time-based or explicit, is allowed to happen. ## What it looks like in practice A concrete real-world pattern: an image-processing pipeline storing uploads in an `S3` bucket configures an S3 Lifecycle rule to expire objects seven days after creation, well past the pipeline's typical processing time of minutes to low hours, giving ample buffer for retries and dead-letter reprocessing. Consumers additionally delete their own objects immediately after successful processing as a cost optimization, since S3 storage cost, while cheap per gigabyte, is not free at scale, and shaving days off the retention of successfully-processed objects meaningfully reduces steady-state storage cost when processing millions of objects a day. The combination -- application-level deletion as the fast path, bucket lifecycle policy as the guaranteed backstop -- is what prevents both premature growth and the silent unbounded accumulation described in the scenario.