Why does normal paid use of a content-moderation API also hand the caller a labelled dataset?

## The two halves of a supervised dataset Every supervised training set is a pile of pairs: an input, and a label attached to it. The two halves have wildly different prices. The **inputs** are close to free. For a text task, unlabelled text exists in enormous quantity — forum posts, marketplace listings, product reviews, scraped pages, synthetic variations of any of these. A competitor who wants a million pieces of text that look like the traffic a real forum operator sees can obtain them without asking anybody's permission and without a budget line. The **labels** are where the money is, and on a policy task like content moderation the gap is extreme. A label for "is this harassment" is not a fact sitting in the world waiting to be recorded. It is the output of a written policy document, an annotator pool trained on that document, a quality process, and an adjudication path for the cases two annotators disagree on. That apparatus is expensive to build, expensive to run, and it *is* the differentiated asset — two vendors with the same architecture and the same crawl differ almost entirely in how their annotation guidelines resolve the hard middle. ## What a prediction endpoint actually sells A paid classification API answers a request by attaching a label to an input. Stated that way, the security reading is immediate: the endpoint is an annotation service that bills per row. The customer brings the input half, which they had already; the seller supplies the label half, which is the half nobody can cheaply make. A caller does not have to do anything unusual to accumulate the result. Ordinary integration already logs requests and responses — for debugging, for caching, for their own analytics, for their own audit trail. After a few months of ordinary production traffic, that log **is** a supervised dataset: their text, your annotations. No scraping, no undocumented field, no error-message spelunking, no terms violation. The published contract, used as published. This is the entry point to what the literature calls query-based model extraction, and the important structural point is that the first stage of the attack is indistinguishable from being a customer. ## What they hold, precisely Overclaiming here is the classic way to lose an argument in a design review, so state the disclosure exactly: - They hold **annotated rows** — their inputs with your model's scores attached. - The corpus covers **only the region of input space they queried**. Behaviour on text they never sent was never observed. - They do **not** hold your parameters. Nothing in a returned score pins a weight value. - They do **not** hold your architecture, your training corpus, or your policy document — only the behaviour those things produce, on their inputs. - They do **not** yet hold a model at all. Fitting one is still ahead of them, and it costs compute. What has genuinely moved is the annotation, and with it the operational form of the policy judgment encoded in the annotation. That is usually the asset worth defending, which is why the disclosure matters even though it is not "the model was stolen". ## Intent does not change the disclosure A caching integrator, a customer building an offline evaluation set, and a competitor building a rival product all end up with the same artefact. The response contract decides what leaves the building; intent decides only whether anybody assembles it into something. That is why this is an API design conversation rather than an abuse-investigation one — you cannot make the disclosure conditional on motive, because motive is not a field in the request. ## Why interviewers ask this Because the wrong answer is comfortable and common: *the model is ours, the weights are private, we are just returning predictions.* The candidate who sees that a prediction is a **label** — supervision, not merely output — has the frame that every later question in this area depends on: what the response body contains is a security decision, and it is made by whoever designs the schema, usually with no security person in the room.