Some external configuration stores support dynamic refresh, where a running service instance picks up a changed value without a redeploy or restart. Describe two different mechanisms services use to detect config changes - one push-based, one poll-based - and explain what can go wrong when a config change is rolled out to a fleet of hundreds of instances.

## What dynamic refresh adds Dynamic refresh solves a narrower problem than the base **External Configuration Store** pattern: not just 'load config at startup' but 'pick up a config change while already running, without a restart.' This matters because restarting hundreds of instances to change one timeout value is slow, disruptive to in-flight requests, and turns a config change into a mini-deployment with all the risk that implies. There are two broad mechanisms for detecting that a change has happened. ## Poll-based refresh **Poll-based refresh** has each running instance periodically ask the store 'has anything changed for me?' - typically on a fixed interval (every 15-60 seconds is common) or with a lightweight conditional check using a version number or `ETag` so the instance can cheaply detect 'nothing changed, skip the reload' without re-parsing the full config payload every time. Spring Cloud Config's `/actuator/refresh` endpoint combined with a scheduled poller, or a sidecar that periodically re-reads a mounted Kubernetes `ConfigMap`, are common implementations. Polling is simple to reason about and inherently **self-healing**: if one poll is missed due to a transient network blip, the next poll a minute later will still pick up the current state, so there's no permanent desync from a single dropped message. The cost is twofold: - **propagation delay** - a change can take up to a full poll interval to reach every instance - **load on the config store** that scales linearly with fleet size and poll frequency, since every instance independently hits the store on its own schedule ## Push-based refresh **Push-based refresh** instead has the store actively notify subscribed instances the moment a value changes, using a long-lived connection or subscription mechanism: | Mechanism | Shape of the subscription | |---|---| | Consul's blocking queries | an HTTP long-poll that returns only when the watched key's index changes | | etcd's native watch API | a streaming gRPC subscription | | a message bus like Kafka | config changes are published as events that a lightweight config-sync sidecar consumes and applies locally | Push mechanisms cut propagation delay to near-instant and avoid the load of constant polling, but they trade that for more operational complexity on the store side: - it must track potentially thousands of open watch connections - it must handle client reconnection gracefully after a network blip - a client that misses a watch event while disconnected needs a way to catch up, usually by falling back to a poll or fetching current state on reconnect - it must avoid a thundering-herd reconnect storm if the store itself restarts and every client reconnects simultaneously ## The window of inconsistency The real danger at fleet scale isn't either mechanism individually - it's the window of inconsistency both create. Because instances receive and apply a change at slightly different times (staggered poll schedules, network jitter, GC pauses delaying event processing, or simply the store notifying subscribers in some non-atomic order), there is always some period during a rollout where part of the fleet is running the old value and part is running the new one. For most settings this is harmless - a log-level change taking effect gradually across the fleet is fine. But for settings that must be **atomic across all instances** - a schema-related flag, a protocol version toggle, a value that two collaborating instances both need to agree on simultaneously (like a distributed lock timeout or a consistent-hashing ring parameter) - this **split-brain window** can cause genuinely hard-to-reproduce bugs: - two instances disagreeing about how to interpret the same request - a coordination protocol breaking because half the fleet expects one behavior and half expects another These bugs look like intermittent race conditions and are notoriously hard to diagnose because the 'cause' (an in-flight config rollout) is transient and often already finished by the time someone investigates. ## The standard mitigations The standard mitigations are: 1. Treat non-atomic-safe config changes with the same caution as a **code deploy** - canary the change to a subset of instances first, monitor, then roll out gradually rather than pushing to 100% of the fleet at once. 2. Make the reload operation itself **atomic and side-effect-free within a single instance** - swap a config object reference rather than mutating fields in place, so a request never sees a half-updated config mid-reload. 3. For genuinely fleet-wide-atomic settings, avoid dynamic refresh entirely and require a coordinated restart or a feature-flag-style gate that's explicitly designed for synchronized cutover instead.