After adding `--read-only`, a Node.js queue worker dies with EROFS. How do you find every path it writes?

## Why one-error-at-a-time fails The tempting loop is: start with `--read-only`, read the EROFS in the log, add a tmpfs for that path, repeat. It ends badly, because a container only reveals the path it needed *this second*. A queue worker that writes `/tmp` on every message may write `/var/log/worker/poison.log` only on a malformed payload and a lock file under `/run` only during a graceful shutdown. Patch by patch, you ship a container that runs fine for eleven days and then dies at 03:00 on the one code path nobody exercised. Build the full inventory first. ## Step 1 — get the inventory with `docker diff` Run the worker in its normal (writable) configuration and exercise it deliberately: a batch of ordinary queue messages, a malformed one, a downstream timeout, a config reload, a SIGTERM shutdown. Then: ```bash docker run -d --name scorer-probe fraud-scorer:1.4 # replay 250 messages, including 3 poison payloads, then stop it politely docker diff scorer-probe | grep -v '^D ' | sort ``` `docker diff` compares the container's writable layer against the image and prints one line per path: `A` added, `C` changed, `D` deleted. That output *is* the answer to "what does this thing write". Two caveats to state out loud in an interview: it reports only the writable layer, so writes into a bind mount, a named volume or a tmpfs are invisible to it; and it lists directories that were merely touched, so read it with judgement rather than mounting a tmpfs on every line. ## Step 2 — classify each path, do not reflexively mount it For a 1.7 GB Node.js image the diff usually falls into four buckets: - **True scratch** — `/tmp`, `/run`, a per-request working directory. Mount a tmpfs with an explicit `size=` and `noexec,nosuid`. - **Runtime cache** — an npm or transpiler cache, a compiled-template directory, a `.cache` folder under the app root. Usually a tmpfs, but ask first whether it should be warmed at build time instead: a cache that is rebuilt from scratch on every start is costing you startup latency, not saving it. - **Durable state** — an offset file, a dead-letter spool, anything whose loss changes behaviour after a restart. Named volume. Discovering this is a bonus: you have just documented the service's real state, which was previously hiding in a disposable layer. - **Things that should never have been runtime writes** — the scoring model the worker downloads at boot, a generated asset bundle, a config rendered from a template. Move them into the image. This removes the write, makes start-up deterministic, and means every replica runs identical bytes. ## Step 3 — size the tmpfs from evidence RAM-backed scratch is charged to the container's memory, so `size=` is a real decision, not a formality. Measure the peak with `du -sh` on the probe container's directories after the load run, then set a ceiling with headroom — for a worker whose staging directory peaked at 31 MB, `size=96m` is honest; `size=2g` quietly hands back the ability to fill host memory. Remember tmpfs starts empty on every start and is private per replica, so anything that must be shared or must survive cannot live there. ## Step 4 — prove it under load, not at startup Re-run with `--read-only` and the new mounts in staging, and drive the same full workload — including the error paths — for long enough to be convincing. Watch the logs specifically for `EROFS`. Keep an alert on that string for the first production days; it is a distinctive, unambiguous signal that you missed a path, and it costs nothing. ```bash docker run -d --name scorer \ --read-only \ --tmpfs /tmp:rw,noexec,nosuid,size=96m \ --tmpfs /run:rw,noexec,nosuid,size=6m \ --tmpfs /app/.cache:rw,noexec,nosuid,size=64m \ -v scorer-spool:/var/lib/scorer \ fraud-scorer:1.4 ``` ## What not to do Three answers mark a candidate down. Dropping `--read-only` at the first failure abandons the control for a problem that takes an afternoon to solve. Bind-mounting a writable host directory over a large part of the tree gives back everything the flag took away and adds host coupling. And mounting one enormous unsized tmpfs over `/` is worse than not hardening at all: the filesystem is now writable *and* backed by RAM. ## The wider point Making a service read-only is really an exercise in learning what the service actually does with the filesystem — and most teams find at least one surprise, usually a write that should have been a build-time artefact or an undocumented piece of durable state. The hardening is the payoff; the inventory is the value.