Your engagement covers a chat product whose backend calls a hosted content-moderation service. Do you send your probe corpus straight to the vendor's moderation endpoint, or through the product? What does each choice actually measure?

## Two probe points ask two different questions "What does this filter do with input X?" and "what does this product do when a user sends X?" are not the same question, and a report that mixes their results is one the customer cannot act on. Both runs are legitimate; they buy different things, cost differently, and their numbers must never share a denominator. ## What the direct run buys Calling the vendor's moderation endpoint yourself gives an isolated, reproducible reading per payload, at a known price per call, with no application in the way. Its jobs are narrow and real: - build and sanity-check the corpus; - establish the **control set** of known positives that validates every later session; - and hold a **reference value** for anything the end-to-end run surprises you with. What it does *not* buy is a fixable finding. The customer cannot retrain a black-box classifier, so a direct-run miss lands as "switch vendors" or as nothing — and it ages fast, because the thing it describes can change server-side without notice. ## What the end-to-end run buys Everything the deployment decided, and every one of those decisions is the customer's to change this week. Sitting between the text a user typed and the text the service scores are: - normalisation and encoding; - truncation of long inputs; - the choice of what to send at all (the latest turn, the whole thread, the user's message only); - whether the model's *output* is screened as well as the input; - which categories the app treats as blocking versus merely logged; - and the cut-off, which the vendor's numbers only advise. Then the **error path**: what the product does when the moderation call throttles, errors or times out. If it proceeds unscreened — **fail-open** — then vendor availability has quietly been promoted into a security control, and that is a finding no direct-endpoint run can ever produce, because the behaviour under test belongs to the application, not the classifier. ## The cost of the end-to-end run is attribution A payload that gets through the product could have been: - (a) scored clean by the service, - (b) scored above the line and ignored by the app, - (c) sent in a mangled or truncated form that no longer carried the harm, - or (d) never sent to the service at all. Four different fixes, in four different places. End-to-end traffic alone cannot tell them apart, and a finding that names the wrong layer wastes the remediation as surely as no finding at all. ## How you resolve it Ask for the backend's own logged request and response for the moderation call, and join on a distinctive, benign **correlation marker** you plant in each probe so the two records can be matched without guessing. With that join every end-to-end miss resolves cleanly into a filter miss, a threshold decision, an app-logic decision, or a call that never happened. Without it, you are guessing — and the honest write-up says "layer unattributed" rather than naming one. If those logs do not exist, stop and write *that* up: a security control whose invocations are not logged cannot be operated, tuned or incident-reviewed, and this is usually a higher-severity finding than whatever payload you were chasing. ## Sequencing under a metered budget 1. **Direct first**, small and stratified, to prune payload families that are uninteresting even in isolation and to fix the baseline. 2. **Then spend the larger share end-to-end** on the survivors, where the actionable findings live. The end-to-end calls are also the slower ones — a product round trip includes the model, not just the filter — so wall-clock, not just billing, argues for pruning first. ## How the misleading number appears here A direct-run miss rate reads, to any executive skimming the report, as the product's exposure. It is not: the app may block on a lower cut-off, screen both directions, and catch downstream what the filter missed — or it may fail open and be far worse. Report the two runs as **separate sections**, each with its own named denominator, and never compute a combined figure. If the customer wants one number, the only defensible one comes from the end-to-end run, and it is a rate against your corpus during your window, not a property of the product.