For a send-later service whose delays range from seconds to a year, how do you choose between broker-native delayed delivery and a durable timer store?

## The requirements that decide it A **send-later service** accepts work to run in the future — a reminder next week, a follow-up in 30 seconds, a renewal notice in a year. There are two broad ways to hold that work until it is due, and the choice follows from a handful of requirements: - **Horizon:** the longest delay you must support. - **Precision:** how close to the due time a job must fire. - **Mutability:** whether users cancel, edit or reschedule pending jobs. - **Visibility:** whether anyone needs to list, count or audit pending jobs. - **Volume:** how many jobs are pending at once, and what holding each one costs. - **Operational budget:** how much custom machinery the team can run. ## Broker-native delayed delivery Some **message brokers** let a producer attach a delay to a message; the broker holds it and makes it visible to consumers only when the delay expires. - **Strengths:** almost no custom code, delivery rides the broker's existing durability and consumer scaling, and precision is usually good for short delays. - **Limits that vary by broker:** some cap the maximum delay, sometimes at minutes or days; some support only a fixed set of delay levels; some have no per-message delay, so teams build it from per-delay queues or message expiry; how many pending delayed messages a broker handles well also varies. - **Opacity:** a delayed message is usually not queryable. You typically cannot list a user's scheduled messages, change a payload, or delete one message by ID before it fires. ## Durable timer store A **timer store** is a database table (or bucketed partitions) of jobs with a `due_at`, read by **pollers** that claim and dispatch due jobs. - **Any horizon:** a job due in a year is just a row. - **Mutable and queryable:** cancel is a status change, reschedule is an update, 'show my scheduled messages' is a query. - **Auditable:** the row records who scheduled what, when it fired and whether it was cancelled. - **Costs:** you run pollers, claim logic and partitioning, and precision is bounded by the poll interval. | Requirement | Broker-native delay | Durable timer store | |---|---|---| | Year-long horizon | often capped | yes | | Sub-second precision | often good | limited by poll interval | | Cancel or reschedule one job | usually not directly | yes | | List pending jobs | usually not | yes | | Custom code to run | little | pollers, claims, partitions | ## The tiered hybrid Many large systems use both, each for what it does best: 1. The **durable store** is the source of truth for every scheduled job, whatever its horizon. 2. A **mover** periodically selects jobs due within a **hand-off window** — say the next 15 minutes — and marks them `handed_off`. 3. It publishes each as a broker delayed message (or loads it into an in-memory **timing wheel**) with the remaining delay. 4. The broker or wheel fires it precisely; a consumer checks the store and executes it. 5. Jobs scheduled inside the window at creation time skip straight to step 3, but are still written to the store first. The window must be shorter than any broker delay cap and longer than the mover's worst-case lag, or jobs fire late. ## Cancelling and rescheduling Once a message sits in a broker, you may not be able to remove it. The usual answer is to make the store authoritative and the message a **hint**: ```json { "job_id": "j-81f2", "due_at": "2027-03-01T09:00:00Z", "status": "pending", "version": 3 } ``` - The published message carries `job_id` and `version`. - **Cancel:** set `status = 'cancelled'` in the store. When the message fires, the consumer reads the row, sees the status, and drops it. - **Reschedule:** update `due_at` and increment `version`, then publish a new message if the new time is inside the window. The old message still fires, carries version 2, sees version 3 in the store, and is dropped as **stale**. This tolerates duplicate publications from a mover that crashed mid-hand-off, too, as long as the consumer's check-and-execute step is itself safe to repeat. ## Cost and failure trade-offs - **Pending volume:** holding billions of long-horizon messages inside a broker may be expensive or unsupported; rows in a store are cheap. - **Blast radius:** a broker outage in the hybrid delays only the near-term window; the store still holds everything else. - **Stale-message waste:** heavy cancellation makes the consumer discard many messages; a narrower window reduces that. - **Complexity:** broker-only is simplest when every requirement fits inside its limits — choosing it is a legitimate answer for short, immutable delays.