What is traffic shadowing, and what does diffing the shadowed responses tell you?

### The technique Shadowing sits between a pre-release test and a real release. A copy of production traffic - all of it, or a sampled fraction - is sent to a candidate version deployed alongside the current one. The current version keeps serving users normally; the candidate's responses are collected and thrown away, or handed to a comparator. Nothing the candidate returns reaches a user, so a wrong answer costs nothing directly. **Replay** is the same idea shifted in time: requests are recorded to a corpus and fed to the candidate later, which trades freshness for repeatability and lets you re-run the same corpus after each fix. The value is input diversity. A regression pack contains the cases somebody thought of. A week of production traffic contains the cases nobody thought of: the employee with three concurrent contracts, the pay period containing a leap day, the record with a null cost centre that has been there since a migration in 2019. ### The diff is the oracle - and that shapes what you can learn An oracle is whatever decides that observed behaviour is wrong. Here the oracle is the current version's response. That has a sharp consequence: **shadowing detects change, not correctness**. If the current version has quietly mis-rounded a deduction for two years, the candidate that reproduces the bug produces zero diffs and looks perfect. Conversely a diff is not automatically a defect - if the change was intended, every affected request diffs, and the work is to confirm the population of diffs matches the population you meant to change. So the artefact you want is not a pass/fail but a **classified diff report**: how many requests differed, grouped by the shape of the difference, with an exemplar for each group. ### Side effects: the failure that ends careers The candidate receives real requests. If it shares the primary data store, it will write. If it can reach the outbound integrations, it will send. For a payroll engine that means duplicate ledger entries, duplicate notifications to employees, or - in the worst case - a second disbursement instruction. Before a single request is mirrored you need one of: a candidate wired to an isolated copy of the data, a request context flag that makes every write and every outbound call a no-op sink, or restriction to genuinely read-only endpoints. Read-only endpoints are the natural first target for exactly this reason, and a calculation preview is read-only. A second, quieter side effect is load. Shadowing doubles the request volume hitting shared downstream dependencies unless the candidate is isolated from them too. Sampling a fraction is the usual mitigation, at the cost of covering fewer rare inputs. ### Normalisation: where most first attempts drown Raw response comparison produces a diff rate near 100 percent for reasons that are all uninteresting: - **Timestamps** and other clock-derived values - the candidate is milliseconds behind. - **Generated identifiers** - a new preview identifier per call. - **Ordering** of collections that carry no defined order. - **Formatting** of money and decimals, where one version emits a trailing zero and the other does not. - **Time-of-read skew** - the underlying record changed between the two calls. The comparator therefore canonicalises first: blank out volatile fields, sort unordered collections by a stable key, normalise numeric representation to a fixed scale, and only then compare. Every masking rule is a small loss of coverage, so each one deserves to be written down and revisited - masking a field is exactly how a real regression hides. ### Privacy and the corpus Mirrored and recorded traffic is production data. A replay corpus of payroll requests contains names, salaries and tax identifiers, so it inherits production's access controls, retention limits and redaction obligations. Treating a recorded corpus as an ordinary build asset is a common and serious mistake. ### A worked example A team rewrites the gross-to-net calculator behind a payroll engine and shadows the preview endpoint for one week of a three-week release train. Of 92,600 previews, 287 - about 0.31 percent - differ after normalisation. Grouped by shape, 284 share one signature: the net amount is one day of proration low. Every one of them belongs to an employee whose hire date falls exactly on the first day of the pay period. That is an off-by-one boundary in the proration window, and it is precisely the case the handwritten regression pack did not contain, because nobody writing test data thought to hire someone on the boundary. The remaining three diffs are ordering noise from an unsorted deductions list and are fixed by a normalisation rule. ### When it earns its keep, and when it does not It is strongest for like-for-like replacement of a component with rich input space and deterministic output: a calculation engine, a parser, a pricing or entitlement service, a query layer. It is weak or unusable when the response is legitimately nondeterministic, when the endpoint is inherently write-heavy, when responses are personalised in ways that shift constantly, or when the candidate is not a like-for-like replacement at all - a deliberately new behaviour diffs everywhere and tells you nothing. It also proves nothing about the write path, since you disabled it; the write path still needs a staged rollout with real users to be verified.