What does scheduled.rebalance.max.delay.ms control, and how do you tune it for worker restarts vs. genuine failures?

## What it is `scheduled.rebalance.max.delay.ms` is a **worker-level** Kafka Connect configuration introduced with **incremental cooperative rebalancing (KIP-415)**. Default: **300000 ms (5 minutes)**. It answers one question: *when a worker disappears, how long should the cluster wait before giving its work to someone else?* ## Why a delay exists at all Workers leave the group for two very different reasons: - **Transient**: a rolling restart, a JVM GC pause, a momentary network hiccup, a redeploy in Kubernetes. The worker will be back in seconds. - **Permanent**: the host died, the process crashed and won't restart, the node was terminated. Without a delay, both look identical, and the cluster would immediately move the departed worker's tasks to survivors. For the **transient** case that's wasteful: the worker comes back moments later and the tasks get moved **again** — two rebalances and two task migrations for a blip. With many workers churning (e.g. a rolling restart of a big cluster), that thrash is severe. ## How the delay works When a worker leaves: 1. The leader notices the worker's connectors/tasks are now **orphaned**. 2. Rather than reassigning them, it marks them to be assigned **after** a scheduled delay and leaves them **unassigned** in the meantime. 3. A timer counts down up to `scheduled.rebalance.max.delay.ms`. 4. **If the worker rejoins before the timer expires**, it reclaims its original tasks — no migration, minimal disruption. 5. **If the timer expires first**, the leader reassigns the orphaned tasks to the surviving workers. Note it's a **max** delay: if the worker returns early, the wait ends early. ## The core trade-off - **Higher value** (e.g. 5+ min): great for clusters with frequent rolling restarts/redeploys — avoids churn, keeps assignments stable. **Cost:** when a worker *genuinely* dies, its tasks stay **down** (no data flowing for its source/sink partitions) for the full delay. - **Lower value** (e.g. 0–60 s): faster recovery from real failures — tasks restart on survivors quickly. **Cost:** every quick restart or flap triggers move-out-then-move-back churn. `0` disables the delay: orphaned tasks are reassigned immediately (closer to pre-delay behavior, maximizing availability but maximizing churn). ## How to tune it Match it to your **typical worker downtime**: - **Kubernetes with rolling deploys**: set it comfortably above your pod restart/reschedule time (often a few minutes) so a normal redeploy doesn't migrate tasks. But cap it at the longest data stall you can tolerate. - **Latency-sensitive sinks** (feeding live dashboards/alerts): lower it so a dead worker's partitions resume quickly, accepting more churn. - **Large, stable clusters doing frequent rollouts**: keep the default or higher to minimize rebalance thrash. ## Interaction with other settings - It only matters under **cooperative** rebalancing (`connect.protocol=compatible` or `sessioned`); under `eager` there's no scheduled-delay concept. - It works **with** the group membership `session.timeout.ms` machinery: the worker is first declared *gone* by the group coordinator (session timeout / heartbeat loss), and *then* the scheduled delay decides how long its orphaned tasks wait. They're distinct timers serving different purposes. ## Common mistakes - Believing the delay keeps the whole cluster paused — it doesn't; only the **departed worker's** tasks are idle, everything else keeps running. - Setting it very high to 'reduce rebalances' without realizing a dead worker's data flow stalls for that whole window. - Confusing it with `session.timeout.ms` (how long before a missing worker is even considered gone).