Scheduler Agent Supervisor

## Why the pattern exists The Scheduler Agent Supervisor pattern exists because a multi-step operation that spans several independently-owned services or workers cannot rely on a single in-process `try/catch` to handle failure - any one of those remote calls can fail outright, hang, or succeed without the caller ever finding out. The pattern factors the problem into three cooperating roles. ## The three cooperating roles - **The scheduler** owns the definition of the job: an ordered (or partially ordered) list of steps that together form one logical operation, such as 'reserve inventory, charge payment, schedule shipment.' Before dispatching a step, the scheduler writes a durable record of that step's intended state - typically a row in a database table or an entry in a durable queue - so that if the scheduler process itself crashes and restarts, it can read that record back and know exactly where the job was, rather than losing track of in-flight work. - **The agent** is the thing that actually talks to the outside world: it wraps a single remote call (an HTTP request, a message to a worker queue, an RPC to another service) and is responsible only for making that call and reporting the outcome - success, business failure, or 'no answer within the timeout' - back into the durable state. - **The supervisor** is a separate watchdog process (or logical role, sometimes folded into the scheduler process but conceptually distinct) that periodically scans the durable state for steps that are overdue, failed, or stuck, and decides what corrective action to take. ## How one run actually goes Mechanically, a run looks like this: 1. The scheduler writes `step 2, dispatched, agent X, deadline T` to the state store, then invokes the agent. 2. The agent calls the remote service and, on response, updates the record to `succeeded` or `failed`. 3. The supervisor's loop wakes up on an interval, queries for steps whose deadline has passed with no terminal status, and for each one applies a remediation policy: retry the step (same or different agent instance), reassign it to an alternate resource pool, or - if the step cannot succeed and prior steps already produced real-world side effects - trigger a compensating action to unwind them. The key design decision the supervisor embodies is that a **timeout is not the same as a failure**: the remote call may have succeeded but the acknowledgment was lost, so any remediation must be safe to apply to a step that already completed, which is why agents and the operations they wrap need to be idempotent or the state store needs to de-duplicate by an operation ID. ## The trade-off The trade-off this pattern buys is resilience to partial failure and crash-mid-operation at the cost of real infrastructure. You need: - durable, queryable state (not just in-memory retry loops); - a polling or event-driven supervisor loop; - disciplined idempotency on every agent call, because retries and reassignment will eventually cause a step to be attempted more than once. That infrastructure is overkill for a short synchronous call chain that can just use a client-side retry-with-backoff; it earns its cost when the operation is long-running, spans process or machine boundaries, and must survive the scheduler or an agent's host being killed mid-flight. A second cost is added latency for the failure path: nothing gets remediated until the supervisor's next poll cycle notices the missed deadline, so pick that interval as a genuine business trade-off between detection speed and load on the state store. ## Failure modes in production Common failure modes in production include: 1. **The supervisor itself becoming a single point of failure** if it isn't made resilient - running it as a singleton with no leader election or failover means a crashed supervisor silently stops remediating stuck jobs, even though the scheduler and agents are fine. 2. **Timeout miscalibration** is another: too short a deadline makes the supervisor 'retry' steps that were actually still in flight, producing duplicate charges, duplicate emails, or duplicate shipments if idempotency wasn't enforced; too long a deadline leaves customers staring at a hung operation. 3. **State-store contention** is a third - if every scheduler and supervisor instance hammers one hot table for status updates and polling, that table becomes the actual bottleneck of the whole system, which is why production implementations often use partitioned tables or a proper workflow engine's built-in state management instead of hand-rolling one. ## Where you have already seen it A concrete, well-known realization of this pattern is a durable workflow engine such as Temporal (formerly Cadence) or Azure Durable Functions: - the orchestrator function plays scheduler and supervisor together; - activity functions are the agents; - the engine gives you durable history, automatic timeout detection, and configurable retry/compensation policies out of the box rather than requiring you to hand-build the state table and polling loop yourself.