Testing AI-Powered Features

An adjudicable acceptance criterion names one observable property of the response, says where to look for it, and fixes the decision rule in advance, so two testers reading the same response record the same pass or fail.

A criterion with no stated allowance is read as absolute, so the first response that misses it forces an unplanned argument. Agreeing the allowance up front makes a rare miss an expected, recorded outcome rather than a crisis.

Write the evidence at failure time, because a repeat may never reproduce it: the exact input, the material and instruction text given to the generative step, the model identifier and generation settings, the raw output, and a correlation id.

Five layers can own it: the ordinary product code around the generative step, the instruction text sent to it, the material supplied with it, the configuration, and the generative model's own capability ceiling. Attribute only after evidence separates them.

Everything except the generated wording stays exactly assertable: the dispatch path and its inputs, the response's non-text fields, citation rendering, the empty and cut-short states, and permission refusals. Map that surface first, then write looser checks over the text.

Assert one of three outcomes and assert it works: a manual route the person can finish themselves, a queued promise that really arrives later, or an honest stop. A dead end that merely looks handled is a defect.

Three seams exist: replay a response recorded from a real call, return an answer the test author wrote, or make the real call. The first two make a failure point at your own code; the third does not.

A pass rate describes one judged sample of varying output, not the feature itself: re-judge the same cases and the number moves. Report a measured residual error rate per user action, with its sample size, judging rule and split by consequence.

Repeat count follows the success-rate difference you need to detect; three to five repeats separate only coarse differences. A four-in-five rule claims the feature's per-attempt success rate is high, not that any single user attempt will succeed.

Because the three carry different obligations. A must failing means the promise was not kept and the case fails; a should failing is recorded but does not block; a must-never failing stops everything regardless.

Only the raw answer separates a bad answer from good text the product mishandled. With just the derived value, a deterministic parsing, trimming or fallback bug in the team's own code reads as variation in the generative step.

Four states have fixed expected results: the streamed pieces must assemble and terminate cleanly, a length ceiling must show the defined cut-short marker, nothing to answer from must show the defined empty state, and a caller without permission must be refused outright.

Re-running draws another sample from the same varying feature; it changes the reported result, not the feature. Under a pass rule that already tolerates the expected variation, a repeated red is evidence the success rate is genuinely below the bar.

Rule out deterministic differences before blaming any layer: the literal inputs each user sent, the material each account's request assembled, the variant and settings served, and the client each used. Only then consider run-to-run variation.

Derive every criterion from the product's stated promise before any response is read, then adjudicate each response against that list only. Fluency, extra detail and pleasing tone are not criteria, so a well-written answer that omits a promised item fails.

Apply per-field redaction in the writer, so the unredacted form never reaches the store; keep enough shape that the record still explains the failure; set and enforce a retention window; and default read access to the people investigating that feature.

Without the supplied material, the instruction version, the model identifier and the generation settings, an investigation cannot say which wording or model produced the failure, whether a repeat is the same call, or whether a later edit fixed anything.

Run the cheap deterministic checks first: replay the recorded call, compare the shown answer with the returned one, and read the instructions and material exactly as sent. Only then vary one layer at a time and re-measure the failure rate.

It rests on evidence, not exhaustion: a measured failure rate on fixed inputs, and every cheaper layer changed and re-measured one at a time. In place of a fix, record the limit, its conditions, its containment, and when to re-check.

Observe the hand-off between the decision and the action: record the selected action's name and its argument values, then assert those against exact expected values. Pin the request so the case repeats, and assert nothing about the text that came back.

Make the failure cases first-class: run them on every change rather than in an occasional batch, assert something positive instead of the absence of an error, review the fallback view whenever the primary view changes, and watch how often each fallback fires in production.

Assert the retry's price, not only its success: the total wait the person can now face, a hard ceiling on attempts, that only transient failures are retried, and that the extra calls are counted somewhere people read.

Stamp the recording with its capture date and the configuration it was captured under, and refresh whenever any of that changes. Comparing it word for word against a fresh real answer proves nothing, because the real step never repeats itself.

Replay hides any change in what the real step now returns, including a shape the parser cannot handle. An authored answer hides everything the real step actually does. A live call hides shapes it happens not to produce that day.

Classify each wrong answer by what it costs rather than how often it happens: whether the user notices at the time, whether it can be undone, and whether the feature acted or only proposed. Then set a separate tolerance for each class.

Wrong answers are forecast, so one bad output reverses nothing. Write a rate on an observable signal — edits, discards, repeat attempts, escalations — with a review level, a revert level, a time window, a named person watching, and a fallback that already exists.

Pin the generative step to one fixed recorded answer and repeat: if the result still moves, the test is the source. Then read which assertion moved - generated wording varies by design, a stored identifier does not.

Keep the merge gate deterministic and put the rich live cases on a schedule and before a release, with a named owner. Add one minimal real call to the gate so a total outage of the step cannot pass unnoticed.

Compare against the process the feature replaces, not against zero. Establish whether the figure is trustworthy, what each wrong answer costs, whether errors are detectable by the user and by the team, and what reverting takes. Then record the number, its conditions and who accepted it.