Automation Strategy

A case pays back when it repeats often, the behaviour it checks is stable, its expected result can be judged mechanically, and running it by hand is slow or error-prone. High risk and a defect history raise its value further.

Checks that run once, that need a person to judge look, wording or feel, that have no dependable expected result, and that drive an interface still being redesigned stay manual. Each of those costs more to automate than it returns.

Compare authoring plus recurring upkeep and triage against the manual cost per run multiplied by the runs expected before the behaviour changes. The break-even run count, and whether the horizon reaches it, decides. Most estimates omit upkeep.

Rank candidates by expected loss avoided per engineer-hour: severity and defect history, run frequency, manual cost, and expected upkeep. Reserve capacity for upkeep of existing cases before funding new ones, and record what stays manual and why.

Assert the status code, the few fields the case is actually about, and the response shape the contract promises — and on failure paths the error envelope's code. Whole-body equality breaks whenever an unrelated field changes.

Fetch the token programmatically from the token-issuing endpoint at setup using credentials injected from the environment, cache it per test identity, and refresh it before it expires. Never commit a hand-copied token.

Place each assertion where its oracle lives: rules whose evidence is a value in a response belong at the service interface, while only what a person can observe needs the screen. Assert each rule at one level.

Send the identical request twice with the same caller-supplied idempotency key, then assert the observable state changed once — one record, one downstream effect — reading it back through an authoritative path rather than a cached view.

A data-driven test runs one piece of logic over many input rows held in a table or file. Each row should execute as its own named case, so a failure names the offending row instead of the whole test.

The table owns which named actions run, in what order, with what data and what expected outcome. Code owns each action's implementation: how it drives the interface, its waiting, its argument checks and its failure message.

Rebuild at the keyword boundary what the machine stack no longer says: print the keyword invocation path, the step index, the bound arguments, a subject identifier from the system, expected versus observed, and per-row artefacts.

Wait on state that exists only once the work has finished: a persisted record, a status field at its final value, a drained queue. A proxy such as elapsed time or a progress indicator can be true before the work is done.

Scale the poll interval to the cost of one check: small for a cheap local read, larger than the latency of a network call. Derive the timeout from a measured high percentile of the real wait plus headroom.

A blanket raise withdraws the claim that the system finishes in a known time. It hides performance regressions, wrong readiness signals and real races alike. Measure each wait's actual duration, set per-operation budgets from the tail, and fix the signal.

A retry is legitimate when it mirrors a contract the system really offers — at-least-once delivery, bounded eventual consistency, a published idempotent operation — and the case asserts that guarantee. Otherwise the retry is discarding evidence of a race.

A failure should name the case, state expected versus actual in real values, and point at the failing step, with artefacts captured at that moment - logs, a screen or payload capture, the environment - so nobody has to re-run it to learn what happened.

Move it out of the gating lane but keep it running in a non-blocking one, and record the symptom, a named owner, an expiry date and the behaviour now unguarded. Enforce owner, expiry and a size cap in the build.

Treat every case as paying rent in runtime, upkeep edits and triage time, and earning it by covering a risk nothing cheaper covers. Delete duplicates of lower-level checks, change-detector cases and cases guarding removed behaviour - in reviewable batches, with the accepted risk written down.

Wall clock is set by the slowest shard, so balance shards by each case's measured duration from previous runs rather than by name or count, and make shards genuinely independent - no shared mutable data, fixed identifiers, ports or accounts.

Drive the screen only for the behaviour the case is about. Everything before that - accounts, records, settings, a starting state - should be arranged through a faster channel, so the case checks one thing and fails for one reason.

Assert the outcome a person would notice - the value, message or state the screen now shows - and assert its content, not just that an element exists. A case that cannot fail when the feature is broken is not an oracle.

Keep at the screen level only what needs a screen: that the journey wires together and shows the right result. Rule and combination variation belongs where the rule lives, with one screen case proving the path.

An unused inherited setup step costs runtime and reliability: it still runs on every execution, so the case is slower and can fail for reasons unrelated to what it checks.

Setup that breaks is not a failed check. Keeping provisioning in one step before the assertions lets a run report could-not-build-the-premise separately from the-product-behaved-wrongly — two different problems with two different owners.

Default to a fresh driver session per case: it is the only lifetime that guarantees a clean start. Reuse across a class or a parallel worker only when acquisition is measurably slow; one case can then poison the next.

A named target is one indirection point the run resolves once, so the same suite can be pointed anywhere. An address written into a case pins that case to one deployment and must be found and edited when it moves.

Resolve configuration once at startup in one fixed, documented order: built-in defaults, then a checked-in file, then process environment variables, then explicit overrides, each layer replacing the last. A fixed order makes every run's effective settings predictable.

Case count measures how much was written, not how much can be detected. Drafted cases usually re-walk journeys existing cases already walk and check outcomes already checked, so the set of failures the suite can catch may not grow.

A repair hides every change that broke the original anchor: a control that moved, was relabelled, lost its announced name, or was replaced by a different one. The suite reports the flow works while the interface changed.

An outcome assertion checks a fact the system decided or stored on its own. A restating assertion only re-reads what the case supplied or already watched happen. Ask whether it would still pass with the behaviour removed.

Two: what correct behaviour actually is, and which behaviours carry real consequence. A tool drafting from a running system can only describe what it observes, so it will happily record a defect as the expected result.

Delegate the verdict only when the acceptable result is a family you cannot enumerate: free-form text, wording shown to a user, perceived quality. Where an exact value or a structural rule exists, an explicit comparison is strictly better.

Point the system's configured destination at a receiver the run stands up, then read the recorded request back from it. The receiver must be reachable from where the system runs, not only from the machine running the suite.

Assert on a settled invariant, a property that stays true once the flow has finished, such as a terminal status or a final total. Do not assert on a counter or an intermediate status that is true for only a moment.

Publish the identical order message twice — same identity, same body — wait for evidence that the second delivery was consumed, then assert a single visible effect: one row appended, one balance movement, one outbound notification.

Order is promised only inside a scope — usually a routing value such as an account or order identifier, or a single producing path. Read the design to find that scope, then confine the assertion to events sharing it.

Publish the bad message through the same path a real producer uses, give it a payload that fails at exactly one nameable stage, and tag it with a unique identifier so the case can find it wherever it lands.

Assert the specific values a person would lose - the text already typed, the step reached in the flow, the selections made, any running countdown. Checking only that the app reopened proves nothing about restoration.

A long press is a distinct input event that opens behaviour a tap never reaches: a context menu, selection mode, a drag handle. A tap case proves only the primary action, so the held gesture needs its own case.

Handheld users choose when to update, and many devices stop receiving platform updates entirely, so the installed base stays spread across several version lines for years. A suite that runs only on the newest line leaves most real users unverified.

Declare the link as an input: a named profile with fixed added delay, loss rate and bandwidth cap, applied at one controlled point in the request path. Then assert product behaviour under that profile, not raw elapsed time.

Assert three times: while disconnected, that the action is locally accepted and marked pending; at reconnect, that delivery starts unprompted; once drained, that the far side holds the work and the device now matches it.

A derived address is self-describing: built from the case and run identifiers it is unique, so concurrent runs cannot collide, and it names the case and run that sent the email, making a stray email diagnosable on sight.

Match on structure, not on prose. Collect every link target in the message body, keep the one whose address is the activation route, and assert exactly one matched. Never anchor on the surrounding sentence.

Generate a unique value inside the case and get it into the outgoing email — a subject-line tag or a custom header field — then search the shared mailbox for that exact value rather than for any recent message.

One negative assertion over the whole rendered body beats per-field checks: fail if it still contains the message template's delimiters, a bare slot name, or the text an absent value renders as. Then assert the seeded values appear.

Sending is asynchronous: the product only hands the message to a delivery path that queues and often batches it. Acceptance for delivery is not arrival, so a case that checks immediately reads a mailbox the message has not reached yet.