The same build of an AI-powered feature answers one user correctly and another wrongly. What differences do you rule out first?

## Works for me means a difference you have not found yet Two users, one build number, two different outcomes. The instinct is to treat this as evidence about the generative model, because a component that can answer differently each time is an obvious explanation. It is also the last one that should be reached for, because four cheaper and entirely deterministic differences sit in front of it, and each of them is checkable in minutes. The framing that helps is this: a build number describes the code that was deployed, and almost nothing else that decides what a user gets. What actually reaches the generative step is a request assembled at run time from the user's text, their account's data, their assigned variant and their session state. Two people on identical code can be sending materially different requests, and usually are. ## The differences worth ruling out | Difference | How to rule it out | What it would mean | | --- | --- | --- | | The literal input text | Compare the two recorded inputs character by character, not the two complaint summaries | Different questions were asked, and no layer is at fault yet | | The material each request assembled | Read the material as sent for both, and check what each account is permitted and able to reach | The assembly of material owns it: missing, stale, or restricted for one account | | The variant and settings served | Check the rollout assignment, region configuration and model version behind each request | Configuration owns it, and an untested combination is reaching users | | The client and session | Compare client versions, cached answers and locale-dependent formatting | The failure is downstream of the answer, in ordinary product code | | Run-to-run variation of the generative step | Only after the four above match: repeat one identical call many times | There is no user difference at all; you are looking at a rate | The order matters because each row is cheaper than the one below it and can end the investigation on its own. It also matters because rows one to four produce *fixable* findings with named owners, while row five produces a rate that has to be managed rather than repaired. ## Ruling them out cheaply 1. **Get both recorded requests side by side.** Input text, assembled material, settings, returned text, shown answer. Comparing two reconstructions is far faster than reasoning about two reports. 2. **Normalise the inputs and retry.** Send the failing user's exact text on the working user's account, and the reverse. If the failure follows the text, the account is not the variable; if it follows the account, the material or the variant is. 3. **Check what each account can reach.** Permissions and organisation boundaries silently change what material is assembled, and this is the difference most often missed because the code path is identical for both. 4. **Compare the settings actually in force**, not the ones in the repository. Gradual rollouts exist precisely to make two users different, and they do that job perfectly. 5. **Only then repeat one identical call.** If the same account, text, material and settings produce both outcomes across repeats, the difference between the two users was never real; it was two samples from one distribution. ## When there is genuinely no difference Sometimes step five is where you land, and the honest statement is that the feature fails at some rate for everyone and two users happened to sit either side of it. That changes the shape of the work completely. There is no per-user bug to fix; there is a rate to measure, a containment to design, and a decision to make about a feature that is only usually right. It also means the two complaints are one complaint, and treating them as two independent defects will produce two independent, contradictory investigations. ## What the answer changes The practical value of this discipline is that it protects against the two errors that cost the most. The first is attributing a real, fixable defect to run-to-run variation, which closes the ticket, leaves the defect in place, and teaches the team that this feature is simply unpredictable. The second is the mirror image: chasing a per-user cause through days of investigation when the two users differed in nothing at all, and inventing an explanation to fit. Both are avoided by the same cheap habit of reconstructing both requests fully before forming any theory about which layer is responsible.