After a route moved to run at many locations near users, an intermittent failure became hard to diagnose. What did you lose, and how do you compensate?

## What the single region quietly gave you Debugging a server-rendered route in one region leans on things nobody lists as features. There is a machine. It has a filesystem you can write to and read back. It runs a process that lives longer than a request, so you can attach to it, sample it, or let it accumulate state while you watch. Its logs sit in one place in one order. When something fails, you reproduce it against the same code on your own machine and watch it fail again. Moving the route to many locations near users removes all of that at once, and the loss is not gradual — the tooling either applies or it does not. ## What you lose, precisely - **Durable local storage.** Nothing written during a request is guaranteed to exist afterwards, so dumps, trace files and “write it to disk and look later” are unavailable. - **A process to attach to.** There is no long-running server to profile, and nothing you accumulate in memory can be relied on to be there for the next request — it may be handled elsewhere entirely. - **Log ordering and locality.** Output exists only as far as the platform drains it, and it arrives from many locations as independent streams. There is no single file to tail in order. - **Meaningful stack traces.** The code executing is the bundle produced for that target, so a trace names positions in generated, usually minified output rather than the source you wrote. Restoring the mapping is a separate build concern. - **One request per page view.** The document request, a subsequent data request the page makes, and any pre-routing step are separate invocations that share nothing. A single user-visible failure arrives as unrelated fragments. - **Reproducibility.** Local development typically runs a fuller runtime than the constrained one used near the user, so a failure caused by that constraint may only ever appear in the deployed environment — and possibly only at some locations. | the habit | why it stops working | what replaces it | |---|---|---| | tail a log file on the box | there is no box to reach | structured events drained by the platform | | attach a debugger or profiler | no long-lived process | timing and branch data emitted as fields | | read the stack trace | it points into generated output | source mapping restored at the reporting step | | reproduce locally | local runtime is fuller than the deployed one | reproduce by redeploying to the same target | | “it works for me” | the failure can be location-specific | compare by location, not in aggregate | ## Why the symptoms look random An intermittent failure under this placement is rarely random. It usually correlates with something you are no longer able to see directly: a particular location, a particular resource ceiling being crossed on a larger-than-usual request, a cross-region call timing out under load, or a code path that only executes for one class of request. It looks random because the axis it varies along — which location handled it — is missing from your data. Adding that axis to every emitted event frequently turns “intermittent” into “always, in two locations”. ## A practice that works 1. **Log decisions, not just failures.** Emit the inputs the handler branched on — the matched route, the variant chosen, which reads ran and how long each took — so a failure can be reconstructed without re-running it. 2. **Put one identifier on everything a page view triggers,** and include it in every event from every invocation, so the fragments can be assembled back into one story. Propagating that identifier well is its own discipline, but nothing works without it. 3. **Report at the moment of failure to somewhere outside the request.** If the report is not sent before the invocation ends, it does not exist. 4. **Tag every event with its location,** then look at failures by location before looking at them in aggregate. 5. **Keep the surface small.** The less a near-user route does, the fewer ways it can fail in the place that is hardest to inspect. This is the operational argument for keeping such routes request-only, on top of the latency one. 6. **Use placement itself as a test.** Run the same build in the data's region. If the failure disappears, the cause is in the placement — a runtime constraint, a per-invocation ceiling, a cross-region call — rather than in the route's logic.