In an Airflow sensor, what is the difference between poke mode and reschedule mode?

## The failure this question is really about A team puts a `S3KeySensor` at the top of forty DAGs, each waiting for a vendor file that usually lands at 09:00 but sometimes at 14:00. All forty sensors start at 06:00 in the default mode. Every one of them is a *running* task instance holding a worker slot for eight hours. The deployment's capacity is gone; the actual load tasks, and every other DAG, sit queued behind sleeping sensors. Nothing has failed, nothing has errored, and the platform is dead. That is the scenario `mode='reschedule'` was introduced for. ## What each mode does mechanically Every sensor implements `poke(context) -> bool`. `BaseSensorOperator.execute()` drives the loop. **`mode='poke'`** — the default. `execute()` calls `poke()`; if it returns `False`, the process sleeps `poke_interval` seconds and calls it again, inside the same task-instance execution. The task state remains `running` from the first check to the last, so the worker slot, and any pool slot the task holds, are occupied for the full duration. Advantages: no scheduling overhead per check, sub-minute polling is cheap, and anything the sensor computed stays in memory. **`mode='reschedule'`** — after a `False` poke, the task raises an internal reschedule signal, the process exits, and the task instance moves to the `up_for_reschedule` state. Airflow records when the next attempt is due (it stores reschedule records so it knows when the wait started) and the scheduler re-queues the task then. Between attempts, the sensor holds no worker capacity at all. Costs: each attempt is a full task startup — scheduler pickup, process launch, DAG file parse, connection setup — so a `poke_interval` of a few seconds turns into constant churn on the scheduler and metadata database. ```python S3KeySensor( task_id="wait_for_vendor_drop", bucket_name="landing", bucket_key="vendor/{{ ds }}/data.csv", mode="reschedule", poke_interval=300, timeout=60 * 60 * 8, ) ``` ## Consequences you should be able to state - **Timeout spans the whole wait.** In reschedule mode `timeout` is not per attempt; Airflow measures from the first attempt so the sensor still gives up after the configured total wait rather than restarting the clock each time. - **No in-memory state survives.** A poke-mode sensor can cache a client, a cursor or a partial result between checks. A reschedule-mode sensor starts a fresh process every time, so `poke()` must be self-contained and cheap — and, crucially, it should not have side effects, because it will run many times. - **Startup cost per poke.** If the DAG file is slow to import, every reschedule pays that import again. Long parse times plus short poke intervals is a bad combination. - **Log shape changes.** A poke-mode sensor writes one log for the whole wait; a reschedule-mode sensor produces a log per attempt, which people occasionally mistake for retries. ## The neighbouring knobs - `soft_fail=True` makes a timed-out sensor go to `skipped` instead of `failed`. Use it when the awaited thing is genuinely optional — but understand that skips propagate to downstream tasks under the default trigger rule. - `exponential_backoff=True` grows the interval between pokes, which pairs well with a long timeout on something that is either early or very late. - `deferrable=True`, where the provider supports it, is the third option and generally the best one for long waits: the wait is handed to the async triggerer process, and hundreds of waits cost one process rather than hundreds of task startups. Reschedule mode frees the slot but still pays a full task launch per check; deferral pays neither. ## Choosing Wait measured in seconds, sub-minute polling, cheap DAG parse: poke. Wait measured in tens of minutes to hours: reschedule with a `poke_interval` of minutes, or deferrable if available. And if the awaited event has a producer you control, the strongest answer is to remove the wait — have the producer trigger the consumer instead of paying anything to poll.