A team proposes running the end-to-end suite against a nightly anonymized copy of the production database instead of against data the tests create for themselves. How would you evaluate that proposal, and what would you put in place either way?

## Read the proposal as two separate claims The proposal bundles "our test data is unrealistic" with "therefore run against a copy of production". The first claim is usually true and worth acting on. The second is one possible remedy with heavy costs, and it is worth separating them before deciding. ## What production-shaped data genuinely buys - **Volume and distribution.** Factories create the tidy case: three orders, short names, one address. Real accounts have thousands of rows, an account created eleven years ago, a customer with 400 line items. Pagination, virtualised lists, timeouts and slow queries fail on those and on nothing else. - **Shapes nobody would invent.** Names with combining characters, addresses without a postcode, currencies with no decimal places, records written by a schema version that no longer exists, nulls in fields the current code assumes are populated. - **Coverage of the accumulated past.** Every migration ever run left residue. Synthetic data is always born under today's rules. That is a real class of defect and it is worth having a way to hit it. ## What it costs **Privacy and trust boundary.** Copying real customer records into a test environment moves personal data somewhere with weaker access controls, more accounts, and screenshots and traces attached to every CI run. Anonymization is not a filter you run once and forget — it must handle free-text fields, attachments, logs, foreign keys that must stay consistent after masking, and every new column added next quarter. Treat it as production code: owned, reviewed, tested, with a failure mode of "the refresh fails" rather than "the refresh silently ships real names". Also consider that traces, videos and screenshots produced by a failing test now contain whatever was on the page. **Non-determinism.** If the dataset changes nightly, then any assertion that names a record is a time bomb: the customer you asserted on gets renamed, closed or deleted upstream. Worse, you cannot reproduce a failure — the environment that produced it no longer exists. This is the reason a production copy cannot simply replace tests owning their data. **Cost and friction.** Large snapshots are slow to restore, expensive to store, and generally too big for a developer machine, so the suite becomes something only CI can run. That widens the gap between "passes locally" and "passes in CI", which is corrosive. **False confidence in coverage.** Real data covers what customers happened to do, not what the feature must handle. The edge case you most need — the empty state, the just-created account, the maximum-length field — is often absent from production entirely. ## The arrangement that usually wins Split the roles of the data: 1. **Data the tests assert on is created by the tests.** Unique per run, owned, cleaned up. This keeps the suite deterministic and reproducible. 2. **Realistic background comes from the snapshot** (or from a generator tuned to production's distributions). The environment is heavy and lifelike; the assertions still point at rows the test made. 3. **A separate, scheduled realism job** runs against the real-shaped data and asserts only on invariants: key pages render, no console or server errors, headline journeys complete, response times stay inside a band. It never names a record, so a refresh cannot break it, and it catches the class of defect the proposal was really about. If privacy makes copying untenable — a common outcome in regulated domains — the substitute is a **generator calibrated to production**: measure real distributions (rows per account, field lengths, character sets, age of records) and make the factories produce those shapes without any real values. You lose the surprises nobody thought to model and keep almost everything else. ## How to decide, concretely Ask what defect class the team is actually chasing and whether a cheaper instrument finds it. If it is slow queries and rendering at volume, a generated large dataset does it without touching personal data. If it is "our app breaks on data written by the 2016 schema", a curated set of anonymized *pathological* records — hand-picked, checked in, stable — is better than a nightly full copy: reproducible, small, and reviewable. Full production copies earn their place mainly for migration rehearsals and capacity work, which are not the end-to-end suite's job. Whichever way it goes, write down who owns the anonymization pipeline, what happens when it fails, how long a snapshot is retained, and who can access the environment. A proposal without those answers is not ready regardless of its technical merits.