When would you run a Flink job in reactive mode instead of rescaling it from savepoints?

## What reactive mode actually is Normally a Flink job declares a parallelism, and the scheduler demands exactly that many slots before it will run. The **Adaptive Scheduler** (`jobmanager.scheduler: adaptive`) inverts that: it looks at the slots available and decides a parallelism that fits, up to the configured parallelism, then restarts the job at a new parallelism if the available slots change. **Reactive mode** is the configuration of the Adaptive Scheduler that takes this to its conclusion (`scheduler-mode: reactive`): the job simply uses *all* registered slots. The configured parallelism — set on the job or on individual operators — is ignored; the scheduler asks for each operator's **maximum parallelism**, which is the only ceiling you still control. Scaling the job means scaling the TaskManager pool — start another TaskManager and the job restarts wider; kill one and it restarts narrower, restoring from the latest checkpoint each time. ## What it does not do It does not decide when to scale. Reactive mode is the *mechanism*; the *policy* is external. On Kubernetes that is typically a HorizontalPodAutoscaler on the TaskManager Deployment, driven by CPU or a custom metric. That gap is the first thing to be honest about in an interview: reactive mode alone is not autoscaling, and CPU is a poor proxy for whether a streaming job is keeping up. It also carries hard structural limits: standalone deployments only, application mode only, and exactly one job per cluster. It does not apply to session clusters or to native Kubernetes/YARN deployments where Flink itself allocates containers. ## What a rescale costs Every rescale is a restart. The job stops, a new ExecutionGraph is scheduled at the new parallelism, and state is restored from the latest checkpoint. Concretely you pay: - the **gap** — no processing during restart, so lag grows exactly when you were scaling because lag was growing; - **reprocessing** of everything since the checkpoint it restores — on Flink 2.3 a resource-driven rescale is timed to follow a completed checkpoint (falling back after repeated checkpoint failures or a maximum delay), so a scale-up replays little, but a scale-down forced by a lost TaskManager is a failover and replays everything since the last checkpoint; - **state reload**, which with an embedded RocksDB backend means downloading and rebuilding state per subtask; small state is seconds, large state is minutes; - **key-group reshuffling**, still bounded by the job's maximum parallelism, which caps how far up any autoscaler can go. Multiply that by a flapping controller and the job spends its life restarting. Any policy needs a stabilization window and asymmetric thresholds — scale up quickly, scale down slowly and rarely. ## The judgment call Autoscaling a streaming job is worth it when three things hold at once: idle capacity is genuinely expensive, the load pattern has a large and predictable swing (a consumer-facing pipeline that is quiet overnight), and state is small enough that a restart is cheap. Miss any one and fixed parallelism wins. A pipeline at a steady rate simply has nothing to autoscale to; a pipeline with terabytes of keyed state pays more in reload than it saves in machines; a pipeline whose cost is dominated by the sink or by a fixed Kafka partition count cannot use extra subtasks anyway. The alternative is unglamorous and usually correct: size for peak plus headroom, alert on consumer lag and busy time, and rescale deliberately from a savepoint during a change window when the measured peak has moved. That is predictable, is easy to reason about during an incident, and costs one planned restart a quarter instead of several unplanned ones a day. ## Choosing a signal If you do autoscale, pick the signal carefully. CPU utilization is what an HPA offers by default and is close to meaningless for a job blocking on external I/O or on state access — a saturated pipeline can sit at 30% CPU. The signals that actually track "am I keeping up" are consumer lag and its derivative, and Flink's own busy/backpressured time per operator. The Flink Kubernetes Operator's autoscaler is built on exactly those: it estimates each operator's true processing rate, computes a target parallelism per operator, and applies it — on current Flink through the Adaptive Scheduler's `PUT /jobs/<job-id>/resource-requirements` endpoint, still a restart but with no savepoint round trip — a far better fit than a generic CPU-driven HPA, and the reason most teams reaching for elasticity today go through the operator rather than configuring reactive mode themselves. ## How to present the decision A strong answer names the mechanism accurately, then refuses to treat it as a default. Reactive mode is a real, narrow tool; the interesting content is the cost model of a restart, the fact that the scaling *policy* is yours to supply, and the observation that for most long-running stateful pipelines the cheapest form of elasticity is not scaling at all.