An input screening model's weights and threshold are private: why can an attacker still map its coverage?

## The wrong answer this corrects The common claim is: *our classifier is proprietary, the threshold is not published, nobody outside knows what model it is, so nobody can evade it.* Every clause is true and the conclusion does not follow. Nobody needs to see a screen that answers. ## A deployed screen is a queryable oracle In a self-serve chat product with no tools, the only thing anybody can do is type a turn and read what comes back. That is enough. Each submitted turn produces one of a small number of outcomes: the turn was stopped before generation, the turn reached the assistant and was answered, or the turn reached the assistant and was declined. Attributing correctly between the last two matters, but the point stands: the system returns a value for every input it is given. A value returned for an input is a **labelled sample**. Fitting a boundary from labelled samples is exactly the problem the screening model itself was built to solve; the only difference is that the person probing pays per label in requests instead of collecting a dataset up front. Nothing about weights, architecture, training corpus or the numeric acting point is required to do it, because none of those are what is being learned. What is being learned is where the decision changes. ## Why coverage is uneven in the first place A screening model learns from examples. The examples cluster: a subject appears many times in the wordings that were common in the data used to build it, and rarely or never in wordings that were not. Anything phrased far from the training distribution scores low, not because the screen decided it was acceptable but because it does not resemble what the screen learned to recognise. The result is a boundary with thin regions, and this is a property of trained classifiers generally rather than a defect in a particular one. That is why the map is worth building: the unevenness is real, and it is not visible from the configuration. ## Information per probe, not probes per hour Not every submitted turn teaches the same amount. A turn deep in a region that is obviously going to be stopped returns a label that was already predicted, and predicted labels carry almost no information. A turn whose outcome you genuinely cannot call in advance splits the space, and one that only just tips over or only just stays under locates the boundary within a much narrower band. A campaign that submits high volume without regard to where the boundary probably is spends far more requests for a coarser map than one that spends probes where the answer is uncertain. This is also why the number is small. A map good enough to be useful is not thousands of samples; a few dozen well-placed ones already tell you which framings the screen recognises and which regions it does not cover. ## What the responses do and do not prove Direction matters, and this is the part that gets marked in interviews. | Observation | What it proves | What it does not prove | | --- | --- | --- | | The turn was stopped | The text scored above the acting point | That the assistant would have refused it | | The turn passed | The text scored below the acting point at that moment | That the content is harmless, or that the assistant answers | | The turn was answered | The assistant produced a reply | That the screen ever fired, or that any stage judged the content | And because the outcome near the boundary is not deterministic, one observation of any of these is a sample, not a label. Points that will be built on need repeating, which multiplies the request cost of everything above. ## What secrecy actually buys It is worth saying plainly, because it is the honest half of the answer. Not publishing the configuration means the first probes are spent discovering things a leaked configuration would have handed over for free: whether there is a separate stage at all, roughly what it reacts to, and how sharply. That is a real price, paid in requests, accounts and elapsed time. It is a cost, not a boundary. The signal is emitted by the running system, and the only way to stop emitting it is to stop answering. ## The half-life A map fits one deployment at one moment. Retraining the screening model, moving the acting point, adding a normalising step before it or changing the assistant behind it all invalidate parts of the map without announcing which parts. So the map is perishable, while the property that produced it — a screen that responds discloses its coverage — is not.