Walk through the claim check pattern step by step for a pipeline where a producer emits a large report and a consumer needs to process it: what exactly happens, in what order, and what does each side own?

## The cast and the one hard constraint The claim check pattern has exactly two participants, a producer and a consumer, plus two infrastructure pieces, a message broker and an external storage system, and the mechanism is best understood as a strict **four-step handoff** with one hard ordering constraint. ## The producer side **Step one: write the payload.** The producer generates or receives the large payload -- a report, an image, a video, a batch extract -- and writes it to the storage system. This is a synchronous call from the producer's point of view: it must wait for the storage system to acknowledge that the write is durable before moving on, because everything downstream depends on the payload already existing by the time anyone tries to read it. The storage system returns, or the producer deterministically constructs, an identifier for what it just wrote: - an object key, - a full URL, - a composite of container name plus blob name, or similar. **Step two: publish the claim check.** The producer constructs the claim check itself and publishes it to the broker. The claim check is not the payload -- it is a small envelope containing the reference from step one plus whatever metadata the consumer will need to safely and efficiently retrieve and validate the payload: - a **content type**, so the consumer knows how to parse it; - a **size**, so it can pre-allocate buffers or reject unexpectedly large fetches; - often a **checksum** or hash, so the consumer can detect corruption or a mismatched object after fetching. This publish happens only after step one's write is confirmed durable -- publishing the reference before the write completes is the single most common bug in claim-check implementations, because a consumer could receive and act on the reference before the payload actually exists in storage, producing a spurious not-found error that looks like a transient failure but is actually a logic bug in the producer. ## The consumer side **Step three: receive the reference.** The consumer receives the small reference message through whatever subscription mechanism the broker offers -- a queue poll, a push subscription, a `Kafka` partition read -- exactly as it would for any other message, since from the broker's perspective this is an ordinary small message with no special handling. The consumer deserializes it and extracts the reference and metadata. **Step four: fetch the payload.** The consumer makes a separate, direct call to the storage system, not the broker, to fetch the actual payload using the reference. This is a genuinely new network hop that doesn't exist in a broker-only design, and it introduces its own failure surface: - the storage system might be unavailable; - the object might have been deleted by a retention policy before the consumer got to it; - the fetch might return corrupted bytes that fail the checksum check. Once fetched, the consumer processes the payload as if it had arrived directly in the message. ## Who owns cleanup A design decision that doesn't always get made explicitly is who owns cleanup: 1. does the producer delete the payload after some retention window regardless of consumers, 2. does the consumer delete it once processing succeeds, 3. or does a separate lifecycle policy on the storage bucket handle expiry independently of both? Each choice has different failure implications, covered in more depth by the pattern's lifecycle and cleanup concerns. ## A worked example A concrete worked example makes the ordering concrete: a video-upload service accepts a 200 MB file from a user, streams it to an `S3` bucket, and only after S3 confirms the multipart upload is complete does it publish a message like `{'videoId': 'abc123', 'bucket': 'raw-uploads', 'key': 'abc123.mp4', 'sizeBytes': 209715200, 'sha256': '...'}` to an `SQS` queue. A transcoding worker polls that queue, receives the message, calls S3's `GetObject` with the bucket and key from the message, streams the video into its transcoder, and on success publishes its own downstream claim check pointing at the transcoded output. Note that the transcoding worker never once receives video bytes through SQS -- SQS carries only the roughly 150-byte JSON reference, while S3 does all the heavy lifting of storing and serving the actual 200 MB of data. ## The reference schema is the contract This division of labor is exactly why the pattern exists: each system does the job it's built for, and the coupling between them is a small, well-defined, easily-versioned reference schema rather than the raw bytes themselves. Because that schema is the only contract between producer and consumer, it is worth treating it with the same care as any other API: - adding new optional fields over time is safe; - but renaming or repurposing an existing field can silently break every consumer still expecting the old shape, which is a maintenance concern that a purely inline payload design would not have surfaced in the same way.