The mounted configuration file changed minutes ago, yet the service still enforces the old rate limit — why?
answer
- two events, not one
- the value at rest versus the running copy
- read once at start-up
- parsed into memory, then never revisited
- re-read, watch, or replace
basics
~20 sDelivery and consumption are separate events. The process parsed the file once at start-up and serves from that in-memory copy; nothing re-reads it unless the process was written to, so the old value stands until it re-reads or is replaced.
solid answer
~50 sChanging a delivered value and a running process using it are two different events, and only the first one happened. The platform made new bytes visible at the mount path; that is where its job ends. The process opened that file once during start-up, parsed it into whatever structure the request path actually consults, and has not looked at the path since — so every request is still evaluated against the copy in memory. Three things can close the gap: the process can re-read on a reload signal, it can watch the mounted file and re-read when it changes, or the instance can be replaced so a new process reads the current bytes at start-up. If the process has none of the first two, replacement is the only path, and until then the old value is simply what the service enforces.
go deeper
Recall that a service reads its configuration when it starts and keeps that copy. Changing the delivered value is one step; making the process read again is a second one that has to be arranged.
Explain why holding a parsed copy is the sane default — parsing cost, derived structures, coherent evaluation — and name the three ways the new value can take effect, with what each one costs.
Show the diagnosis order: bytes on the instance, whether a re-read path exists, whether it re-read and rejected, and whether every replica is in the same state. Be explicit that one restarted instance proves nothing about the fleet.
Frame it as a policy question: which settings are worth supporting a reload path for, and which are cheaper to change by replacing instances. Supporting reload is code your team owns and tests forever.
## An edit and an effect are two events When someone says "I changed the config and nothing happened", the change has almost always passed through only the first of two independent steps. 1. **The value at rest changes.** Someone edits the delivered set — here, the rate-limit table the chat gateway reads — and the platform makes the new bytes available to the instances that reference it, at the path where they are mounted. 2. **The process consumes the new value.** Something inside the running process opens that path again, parses it, checks it, and swaps the parsed result into the structure that the request path reads on every call. The platform owns step one. Making new bytes visible does not reach inside a running process and change a value it is already holding — there is no mechanism by which it could, because the process may have turned those bytes into compiled rules, buckets, pools or lookup tables that only it understands. ## Why a process is holding a copy at all Holding a parsed copy is not laziness; it is the normal and usually correct design: - **Parsing costs.** Opening and parsing a file on every request would put filesystem work on the hot path of a service that answers thousands of requests a second. - **Values get transformed.** A rate-limit table becomes counters, windows and compiled matchers. The live structure is derived from the file, not the file itself. - **Coherence matters.** A request should be judged against one complete table. A process that re-read field by field could evaluate half of a request against the old limits and half against the new. - **Most configuration layers bind once.** Whatever loads settings at start-up typically reads them into immutable objects. Nothing in that path subscribes to the file. So the default behaviour of a well-written service is exactly the behaviour that produces this page: the edit is real, the delivered bytes are current, and the running copy is stale. ## The three ways the new value takes effect | Path | What triggers the read | What it costs | Where it fails | |---|---|---|---| | Re-read on a reload signal | Something external tells the process to reload | Cheap; keeps connections and warm state | Nothing sends the signal, or the handler only reloads part of the state | | Watching the mounted file | The process notices the delivered content changed | Cheap; no external step at all | The watcher misses a whole-set swap, or dies quietly and never updates again | | Replacing the instance | A new process starts and reads at start-up | Expensive: restart cost, lost warm state, dropped long-lived connections | Nothing in the declared workload changed, so the platform has no reason to replace anything | The third row is the one that surprises people: editing the delivered values alone usually leaves the declared workload identical, so a platform that keeps the declared shape running sees nothing to do. ## Reading the symptom in order 1. **Check the bytes on the instance itself**, not the source you edited. If the mount still holds the old content, this is a delivery problem and the process is blameless. 2. **Check whether the process has any re-read path.** Look for a reload entry in its logs or a documented signal. If neither exists, no amount of waiting will help. 3. **Check whether it re-read and rejected.** A process that re-reads, fails to parse and keeps the previous table is a very different bug from one that never looked. 4. **Check more than one replica.** One instance that was recreated for an unrelated reason will be on the new table while the rest are not, which produces inconsistent answers rather than uniformly old ones. ## What this is not - It is not proof that the delivery shape is broken. Whether the delivered copy can change under a live process at all depends on which shape it arrived in; that is a separate question from whether the process re-reads. - It is not a change to the declared workload. Editing values and editing the spec that describes the workload are different edits against different objects. - It is not solved by restarting one instance to "confirm the fix". That confirms only that a fresh process reads the current bytes, which was never in doubt. The useful mental model is a cache with no invalidation: the process cached the configuration at start-up, and every propagation mechanism that exists is just a different answer to the question of who invalidates it.
- The mounted file on the instance still shows the old content. What does that change about the diagnosis?It moves the problem back one step: the process is behaving correctly and the delivered value never arrived. Look at whether the instance references the value set you edited, whether the edit was applied where the instance reads from, and whether the projection onto that instance has refreshed yet. Nothing the process does can help until the bytes under the mount are the new ones.
- The service logs a reload and still enforces the old limit. What is the likely bug?The process re-read the file but did not fully adopt it. Common causes: the new content failed validation and the handler kept the previous table, only part of the derived state was rebuilt, or the request path holds its own reference captured at start-up that the reload never replaces. The fix is to build a complete candidate, validate it, then swap the single reference the request path reads.
A printed price list handed to the cashier at the start of the shift. Updating the master copy in the office is real and immediate, and the cashier still charges yesterday's prices until someone hands them the new sheet.
saying these in an interview costs you the question
- Assumes any updated file is picked up automatically by the process
- Says the platform pushes new values into a running process
- Blames the delivery mechanism when the process simply never re-reads
- Treats one restarted replica as proof the whole fleet updated
- Calls the change 'not applied' without looking at the bytes on the instance