Why does one edited cluster-wide default change a running broker node's behaviour immediately while another waits for that node to restart?

## Two kinds of value, one settings surface An operator edits settings through one interface and reasonably expects one behaviour, but a broker node treats its settings as two different things. - A **live-applying value** is consulted repeatedly while the node serves traffic. The node picks up a new value without stopping — either because the change is distributed to it through whatever holds cluster configuration, or because it re-reads its own local source. Retention bounds, rate ceilings, thresholds that gate a behaviour and most numbers an operator tunes during an incident live here. - A **restart-only value** is read once, while the process starts, and is then baked into structures the process cannot rebuild without stopping: the addresses it listens on, the identity it presents to the rest of the cluster, how its storage area is laid out, the size of a pool allocated up front. Nothing about the edit itself tells you which kind you just made. The write is accepted either way. ## The trap: "nothing happened" is not "nothing was changed" When a restart-only value is edited, the running cluster keeps behaving exactly as before. The natural reading — *the change did not take* — is wrong and is the reason this material is asked. The change is **pending**: it is recorded, and it will be applied by each broker node at the moment that node next starts. That moment is rarely chosen. Nodes restart during a planned roll, but they also restart because a machine failed, because a platform moved them, or because someone was recovering from an unrelated incident. So a pending restart-only change is an instruction left for the worst possible moment: the node comes back **and** comes back different, and the operator handling the failure is now debugging two changes at once, only one of which they know about. The inverse is just as common. A value set directly on a running node, without also changing the source that node reads at startup, disappears the moment the node restarts — the fix that "worked" reverts, silently, at whatever hour the restart happens. ## Blast radius follows the class | Property | Live-applying value | Restart-only value | |---|---|---| | When it takes effect | On the running process, usually within seconds | When that node next starts | | How many nodes move together | Effectively all of them at once | One at a time, in restart order | | What the intermediate state looks like | Brief or none | The cluster runs two configurations until the last node is done | | What backing it out costs | Set the old value back | Another restart of every node already changed | This is why the class matters more than the value. A live-applying change is a cluster-wide event with a fast path back; a restart-only change is a multi-step procedure whose middle is a cluster deliberately running two configurations, and whose reversal is a second pass of the same length. ## Telling them apart before you apply 1. **Ask the software, not your memory.** Platforms publish which values may be changed on a running node, and the list changes between releases — a value that required a restart two releases ago may not today. 2. **Reason from the mechanism when the documentation is thin.** If the value decides something allocated or bound at startup, assume restart-only. If it is a number compared against on each operation, it is probably live. 3. **Prove it in a non-production cluster** by changing it and observing the behaviour it should move, rather than by reading the value back — reading it back usually shows the stored value, which is not the same as the value in force. 4. **Treat an accepted write as no evidence.** The interface acknowledging the change says only that it is stored. ## Where platforms differ - The **split itself is not standard**. The same conceptual setting may be live on one platform and restart-only on another, and several platforms have steadily moved values from the second class into the first across releases. - **Distribution differs.** Some platforms hold settings centrally and push a live change to every node; others expect each node to be told separately, which means a live change can reach nodes at meaningfully different times, and a node missed entirely is a difference nobody sees until it matters. - On a **rented cluster** the class is often invisible: the provider applies the change in its own way, and whether that involves restarting nodes for you is part of what you bought rather than something you control. - Some platforms let you set a value **on the running process only**, explicitly not persisted, which is useful during an incident and dangerous afterwards for exactly the revert-on-restart reason above.