You are filing a model-behaviour report with the provider of a hosted chat endpoint your client builds on. What evidence does that submission need that a report of a deterministic software bug would not, and why?

## Repro procedure versus repro statistic A deterministic software bug report carries a *procedure*: run this input, observe this output, every time. A hosted chat endpoint samples from a distribution, so the equivalent claim is a **rate under stated conditions**, and everything the submission needs follows from making that rate checkable by someone who was not there. The conditions block, and why each entry exists: | field | why the triage team needs it | |---|---| | attempts and successes | a single transcript is unfalsifiable; a fraction is a claim someone can re-measure and argue with | | model identifier as returned by the API | the response records what actually answered; a service may route among variants, so the request only records your intent | | sampling settings (temperature, top-p, max tokens, seed) | a rate at temperature 1.2 and a rate at the service default are numbers about different systems | | full prompt state (system prompt, tool definitions, prior turns) | multi-turn effects vanish when a reviewer replays only the last message | | the success criterion | if a classifier or model judge labelled the outputs, the rule it applied *is* the definition of the rate | | run window and timezone | the endpoint can move under you, so the number has an expiry date | | what you did not vary | it states the claim's limits before someone else overstates them for you | ## What a defensible submission costs Fifty single-turn attempts is fifty calls — trivially cheap. The real bill starts when the effect needs a build-up: a six-turn sequence at 50 attempts is ~300 calls, an automated judge adds one call per attempt, and you generally need a second arm measured at the provider's *default* configuration so the comparison exists before they ask for it. That is realistically 800-1,500 calls for one finding: single-digit to low-tens of US dollars on most hosted endpoints, an hour or so of wall clock against a per-minute rate limit with retries, and — the dominant cost — most of an engineer-day building the harness and hand-labelling a sample of transcripts to check that the judge agrees with a human. ## Where the number misleads **Sample size.** A rate from 20 attempts is nearly uninformative. Five successes in 20 is a 25% point estimate whose 95% interval runs roughly 9% to 49%; 25 in 100 is the same 25% with an interval of roughly 17% to 35%. If the finding's weight rests on "this happens often", 20 attempts cannot carry it, and a provider who re-runs 20 of their own and sees one success has not contradicted you. **The judge.** This is the most common way a submitted rate is simply wrong. Suppose the true rate is 10% and the automated judge agrees with human labels about 85% of the time. Of 200 attempts, ~20 are real successes and ~180 are not; a judge that mislabels roughly 15% of the negatives contributes on the order of 27 false positives, so the reported count is dominated by judge error and the headline rate is inflated well past 20%. Nothing about the transcript is fake — the *scoring* is. Hand-label a random sample, publish the agreement number alongside the rate, and keep the raw run records so the rate can be recomputed under whatever definition the provider comes back with. **Denominator.** Your rate is over deliberately crafted attempts. A reader hears it as a property of ordinary traffic. Write the denominator into the same sentence as the number, every time. **Drift.** The system you measured may not exist next week, in either direction, which is why the timestamp is evidence rather than metadata. ## The "cannot reproduce" reply, and what to check Most non-reproductions are conditions mismatches rather than disagreements about reality: the provider retested single-turn, at the default sampling configuration, with no system prompt, at a smaller attempt count, and scored a hit under a stricter harm definition. If your submission already stated all five, the exchange becomes a negotiation over the rate and the criterion instead of an argument about whether anything happened. Before sending, check: are attempts and successes both present; is the identifier the one the API returned; is the scoring rule disclosed in words a stranger could apply; did you sanity-check the judge against human labels; is the attempt count large enough for the claim you are making; and have you re-run recently enough that the number is still about the live system? Withhold the copy-paste operative text — the class, the conditions, the rate and the criterion are what make the report actionable, and the payload itself is not needed for triage.