How does recovering a model's parameters differ from training a copy on its replies?

## Two different things get called 'stealing a model' Take a concrete target: a small equipment-failure-risk regressor sold to industrial customers over turbine and pump telemetry. A handful of engineered sensor features go in; one risk score comes back as a real number. There is no class label anywhere in the contract — the product is the number. An attacker with an ordinary paying account has two quite different things they might do with that endpoint. ### 1. Fit a stand-in Buy a lot of predictions. Treat every pair of submitted features and returned score as a labelled training example. Train your own model on that set. This is the attack the extraction literature mostly teaches, and what it produces is a statistical approximation: it agrees with the target where the queries landed, its agreement decays as you move away from them, and its internals are the attacker's own — their architecture, their weights, their training run. It is a look-alike, and its quality is bought with query volume and with how well the queries covered the space that matters. ### 2. Solve for the parameters This is a categorically different move, and it is the one this topic is about. A network whose units are piecewise-linear computes a function that is linear on each of many regions of the input space and *bends* at the surfaces where a unit switches between its two linear pieces. Those bends are not noise in the function; each one is placed and oriented by the weights and bias of a single unit. So the location of a bend is a statement *about a parameter*, and it is an exact one. An attacker who can read the returned number precisely enough can tell that the function has bent — the change in the reply as the input moves stops being consistent with a single straight piece. Each bend they can pin down is an equation. Collect enough independent equations and the first layer's parameters fall out; with the first layer known, the layer behind it becomes reachable in the same way, and the solve proceeds inward. Nothing is being fitted here. The literature calls this class of result cryptanalytic, or functionally-equivalent, extraction. ## What the solve requires, and what it therefore is not - **Precision, not volume.** The evidence of a bend lives in the low-order digits of a real-valued reply. A top-1 class label carries essentially none of it; a score reported to two decimals erases it. The vantage this attack needs is *many significant digits of a returned number*. - **A piecewise-linear model.** The whole method leans on the function being made of flat pieces with sharp joins. - **A small model.** Both the number of queries and the arithmetic precision needed climb steeply with width and with depth, because a deeper unit's imprint on the final output is fainter. Published recoveries are of networks with on the order of thousands of parameters, not of production-scale ones. The query volume is a detail worth noticing: a solve against a small regressor is a few thousand ordinary-looking calls from one ordinary account. That is the opposite of the noisy, enormous harvest people picture when they hear 'model stealing', and no usage report has anything interesting to look at. ## What you actually walk away with Parameters — but only up to symmetries that leave the computed function untouched: hidden units within a layer can be permuted, and a unit's incoming weights can be scaled up if its outgoing weight is scaled down by the same positive factor. So the recovery identifies an equivalence class, not one canonical weight matrix. Functionally that costs the attacker nothing: every member of the class computes exactly the same replies. ## Why interviewers ask this The reflex answer is 'extraction only ever gets you a behavioural look-alike; the real weights are never recoverable'. That is a fair description of the common attack and a false claim about what is possible. It is not a theorem. What keeps a large deployed model out of reach is **cost and precision, not secrecy of the weights** — and 'not affordable at our size' is an honest statement with an expiry date on it, where 'impossible' is just wrong. A candidate who states the limit as an economic one, and names what would move it (a smaller model, more digits in the reply), is doing the job the question is testing.