An engineer proposes building a delayed-job scheduler by writing a Redis key with a TTL and running the job when its 'expired' keyspace notification arrives. What are the failure modes of that design, and what would you build instead?

## Why this design is so tempting It looks elegant: `SET job:9 payload PX 60000`, subscribe to `__keyevent@0__:expired`, and the notification is your timer. There is no polling loop and no extra data structure. The design is also easy to demo, because with one worker on a quiet instance it works perfectly - which is exactly why it reaches production before its flaws surface. ## The delivery model is the whole problem Keyspace notifications are delivered over Redis Pub/Sub, which is fire-and-forget with at-most-once semantics. The message is written to whichever subscribers are connected at that instant and then it is gone: there is no log, no offset, no acknowledgement, no redelivery, and no way to ask for what you missed. Every property a scheduler needs is absent. Concretely: **Consumer downtime loses jobs.** Any deploy, restart, crash, or network blip is a window in which every event published is lost permanently. Nothing in Redis knows a job was due, because from Redis's perspective the key simply vanished on schedule. **Crash after receipt loses the job.** The consumer got the message, started work, and died. There is no unacknowledged-message list for Pub/Sub, so nothing will hand the work to anyone else. **Slow consumers are disconnected.** Pub/Sub output for a subscriber is buffered, and a subscriber that does not drain fast enough hits the configured client output buffer limit for the pubsub class and is dropped by the server. The failure is silent from the publisher's side, and looks to the consumer like an ordinary disconnect - jobs disappear precisely when the system is busiest. **Every subscriber gets a copy.** Pub/Sub is broadcast, not a work queue. Run three worker replicas for availability and every job runs three times. Bolting a lock on top means each worker now needs an atomic claim, at which point you have built the harder half of a queue anyway and still have no redelivery. **No payload and no value.** The message carries the key name only, and the key's value has already been deleted, so the job's arguments must be stored under a second key or in another system - adding a second failure mode where the payload survives but the trigger is lost, or the reverse. **Timing is not exact.** The event is published when the key is physically removed, which may lag the TTL when the key is untouched and many keys carry TTLs. A scheduler built on it is not even punctual on the happy path. **Placement matters.** Expiry is driven by the primary in a replicated setup, so a listener attached elsewhere depends on propagation; and because notifications use ordinary non-sharded Pub/Sub, in Cluster they cross the cluster bus, which is bandwidth you are spending on a mechanism that still cannot guarantee delivery. ## What to build instead **Sorted set of due times.** Store each job as a member of a sorted set whose score is the Unix timestamp when it should run: `ZADD jobs:due <due-ts> <job-id>`, with the payload in a hash. Workers poll on a short interval, read the members whose score is at or below now, and *atomically claim* them so exactly one worker gets each - a small Lua script that reads and removes in one step, or a move into an in-flight set with a claim timestamp so abandoned jobs can be reclaimed after a timeout. The state is in the dataset, so it survives restarts, is covered by persistence and replication, and is inspectable: you can count overdue jobs, requeue, and audit. **A durable log with acknowledgements.** If jobs are due immediately rather than at a future time, an append-only stream with consumer groups gives per-consumer delivery, retained history, and explicit acknowledgement of each entry, so an unacknowledged job can be claimed by another worker. It solves the loss and duplication problems that Pub/Sub cannot; scheduling into the future still needs the due-time index above. **Or a real job system.** If retries, backoff, visibility timeouts, and dead-letter handling are requirements, that is a queue product's job, and rebuilding it on top of expiry events is a poor trade. ## The one-line rule Keyspace notifications are a *hint* that something happened, useful for cache invalidation and monitoring where a lost message costs a stale entry or a missing data point. They are not a delivery mechanism for work that must happen.