Test Data & Environments

Protection describes what an actor can reach, not what a file looks like. A copy whose stand-ins resolve through a mapping the same credentials reach still holds real people: copy plus mapping is the original data split in two.

Measure what depends on each method, then convert the mother into thin named methods over per-area builders so tests state their own preconditions. Migrate incrementally: new tests use builders, old ones move as they are touched.

Run the case alone and read what it assumes already exists, usually a record, counter or cache an earlier test left behind. Confirm the coupling by shuffling with a recorded seed, then bisect the predecessors to find the one that supplies it.

A field's name says what it holds, not what the tests read from it. Choosing by name routinely destroys the property a case depends on - a boundary, an ordering, a format - while leaving untouched fields needlessly realistic.

A constant column is unarguably private and destroys the column's shape: distinct values, spread, lengths, how many rows a filter returns. Cases that sort, group, deduplicate or search that field then pass or fail for reasons unrelated to the code.

Containment becomes an inventory, not a delete. Enumerate every copy - evidence store, ticket attachment, cache and its mirrors, exports, messages that quoted it, local downloads - remove the ones you can reach, and record the rest as residual exposure.

Nothing forces two independently owned jobs to stay aligned. A changed secret, an upgraded rule, a different normalisation or a local field patch on one side makes the same real value produce two substitutes, and the copies quietly stop joining.

Being unusual is itself an identifier. The largest account, the only customer in a small region, the row ten times the median: those are recognisable by size or rarity whatever the transformation did to the values, because rank and isolation survive it.

Re-record real traffic on a schedule and diff it against the canned responses, validate every canned response against the provider's published schema in the pipeline, and keep a small tier of checks running against the real service so the belief is re-earned.

Measure the drift first — effective settings, schema version, component versions, permission grants — then rebuild from the declared definition. Whatever fails to come up is the list of uncaptured manual fixes. Then refresh on an announced schedule.

A manufactured dataset contains only states today's code can produce, but the live store also holds residue from removed features, skipped migrations and manual corrections. Read paths still meet those rows, so a representative of each must be planted deliberately.

The value used to join the parts, presence on every side, the lifecycle state, and every attribute more than one service copies. Generate all views from one description, then reconcile a sample after loading.

Everything at the ends of the distribution: parents with no children at all, and the rare parent with thousands. A flat two-per-parent set never reaches empty-list handling, paging past the first page, chunking, truncation or tie-heavy ordering.

Intent. A rule set states in readable text what the data must satisfy and what was deliberately left unconstrained, and a diff shows what a change altered. A fitted generator exposes only its output, so review moves there.

Output from a generator fitted to protected records is derived from those records. It can re-emit a near-copy of a rare individual, and it reproduces the same rare attribute combinations, so it stays potentially identifying until an assessment says otherwise.

The route back stops being exceptional. Stand-in values are resolved routinely and by automation, the mapping is copied into every new environment, real values reappear in captured output and defect records, and nobody can list which fields stay reversible.

Split by what each case is evidence for. Logic cases may use a substitute; cases asserting a query, index, constraint, migration or concurrency need the real engine. Hold the split with a stated feedback budget and a periodic replay against both.

Residual risk is never zero, so the job is to bound it and name an owner. Run the checks -- unique combinations, small groups, extremes, unstructured content -- write down what survived, and have an accountable person accept it explicitly.

Pin them in the blocking run, where a failure must reproduce before it blocks anyone, and vary them only in a separate scheduled run that prints what it chose. Total pinning finds no latent assumptions; unmanaged variation reads as flakiness.

Decide from the dataset's job, not the technique: authored rules when stated invariants and explainability dominate, a generator fitted to real records when production-like shape does. Layer them before forking, and run two producers only with a named owner.

A checked-in file of records is compact and reusable, but it separates data from the case that needs it, couples every case that loads it, and rots as the schema moves. Data built in the case stays owned and visible.

Everything that depends on delivery not being exactly once: consumer idempotency, acknowledgement and retry, backoff, dead-lettering after repeated failure, ordering after a redelivery, and reprocessing when a consumer restarts. Handing the object straight to the handler also hides serialization and shared-mutation defects.

Tier it. Keep ordinary run evidence for a short default window, hold longer only what is attached to an open investigation, limit reads to the people investigating and record those reads, then shrink the whole problem by redacting as the evidence is written.

Substitutes come from a finite output space, so two inputs can land on one value - a narrow field, a truncated result, a small lookup set, or two spellings normalised into one. Downstream, two customers silently become one.

Calendar arithmetic breaks where ordinary assumptions fail: 29 February, the daylight-saving days that are 23 or 25 hours long, and a year end where week numbering diverges from the calendar year. Choose those dates deliberately instead of testing on today.

Assert it in the build. A check run after every dataset build resolves each planted label, confirms the row still has the characteristic it was planted for, and fails the build when one is missing or has quietly become ordinary.

Rarely, and never as a standing capability. Establish first that the record's shape cannot be manufactured, then resolve one value under a separate owner's approval, record who asked and why, and keep the resolved value out of the dataset.