skip to content

Why does a perturbation radius fail to describe a loan applicant editing their own application?

level: middleimportance: should knowfreq 50%

answer

  1. who writes the record being scored
  2. no original to stay close to
  3. nothing compares the form against reality
  4. one keystroke, any permitted value
  5. the limit is a price per field

basics

~20 s

An applicant does not nudge a stated income by a fraction of a percent; they type a different number. A radius assumes a true input to stay near, and a self-authored form has none. The limit is per-field cost.

solid answer

~50 s

A perturbation radius encodes an adversary who must stay imperceptibly close to a real input, because a human or a sensor would otherwise notice the change. Nothing enforces that on an application form. The applicant is the author of the record; the only original is the truthful version, and at decision time nobody is holding the two side by side. A stated turnover goes from one value to any other in a keystroke, and the model reads the second as an ordinary row. So a claim that a credit or onboarding model is 'accurate under small feature perturbations' establishes almost nothing about applicants. The honest budget is per field and economic — free to retype, costly to make true, or bound to an attestation the subject cannot forge — and designing against it means deciding which columns you consume and how much weight a free one may carry.

go deeper

for a junior

Know that on a form the subject fills in, the interesting change is a different value typed in, not a tiny nudge, and that nothing automatically keeps a declared number close to the truth.

for a middle

Explain what a bounded-change threat model is actually encoding — an observer or sensor that would notice — and show why that mechanism is absent when the subject authors the record.

for a senior

Be able to read a robustness claim on a decisioning model and say which protocol produced it and which adversary it describes, rather than accepting the headline number.

for a principal

Own the framing your organisation adopts: if teams keep quoting perturbation robustness for self-authored inputs, the threat model in your review templates is wrong and the numbers it produces are reassuring nobody correctly.

## Where the radius picture comes from The familiar way to state an evasion threat model is a norm and a radius: the adversary may change the input, but only within a small ball around a real one. That constraint is not arbitrary. It encodes a specific real-world limit — a person looking at the image, or a sensor capturing it, would notice a large change, so the attack is only interesting if it stays small. The ball is a stand-in for an observer who compares. ## Why it does not transfer to a self-authored record On a loan or merchant application, three of the ball's assumptions all fail at once. **There is no original.** The adversary is the subject of the decision, and they write the record. The only 'true' version is the honest one, and it is never submitted, never stored, and never compared against what arrives. **Nobody is comparing.** In the image setting the constraint is enforced by perception. Here, unless a field is checked against an attesting source, nothing at decision time holds the submitted value against reality. A number that is wrong by a factor of ten is still a syntactically ordinary number. **The moves are not small.** An applicant does not shift a declared turnover by 0.01. They select a different category from a dropdown, or type a different figure. The change is discrete, large and deliberate. The result is that 'we cap the perturbation budget, so an adversary can only move the features a little' is a category error in this setting, not a conservative assumption. There is no mechanism doing the capping. The sentence describes a threat model borrowed from somewhere else. ## What replaces it The adversary optimises over a *feasible set* defined by prices, not over a neighbourhood defined by distance. The feasible set is bounded by three real things: - **Format and range validation** — the value has to be a value the form accepts. - **Internal consistency** — declared volume against declared category against declared employee count, if anything actually cross-checks them. - **Attestation** — the fields somebody else asserts, which cost a fraud against that party. Inside that set, the cheapest point may be nowhere near the applicant's real profile. That is precisely the difference between a price and a radius: a price says *how much it costs to get there*, a radius says *how far you may go*, and the cheapest useful lie is often the largest one. ## What this does to a robustness claim If a report says a tabular decisioning model retains high accuracy under small perturbations of every feature, ask which fields moved, who sets them, and what a change costs the subject. Perturbing all columns by a small fraction models an adversary nobody has: a rounding error with intent. The number that matters is what happens when a single self-declared column is set to a value its author chose freely, and how much of the score that column can move on its own. A high number under the first protocol is not evidence for the second, and quoting it as if it were is the direction-of-claim error this whole area turns on. ## Where a distance constraint still belongs on tabular data It is not that norm balls are wrong everywhere off images. They are the right description wherever a value is *measured* rather than *declared*, or wherever something downstream would notice: a sensor reading, an aggregate the subject cannot address directly, a value cross-checked against a record the subject does not control. The test is simple — is there an observer or a source that would flag a large change? If yes, a bounded-change model is describing something real. If the subject sets the number by assertion and nothing compares it to anything, the radius is describing an adversary who does not exist. ## The boundary Choosing the norm and the radius, and reasoning about what a given pair silently permits, is a separate topic. So is the discrete case where an artefact must still function after editing. The point here is narrower and prior to both: on a record written by the person being judged, neither a norm nor a radius is the constraint in play, and the constraint that is in play is denominated in money.

  • Is there any constraint at all on what an applicant can submit?
    Yes, but none of them are distance constraints. Format and range validation, internal consistency across declared fields, and cross-checks against attested sources together define a feasible set of records. The adversary optimises inside that set. The cheapest point in it can be arbitrarily far from their real profile, which is exactly why a neighbourhood around a true input is the wrong picture.
  • Where does a bounded-change threat model still fit on tabular data?
    Wherever the value is measured rather than declared, or wherever something downstream would notice a large move: a sensor reading, an aggregate the subject cannot address directly, a figure cross-checked against a record they do not control. If an observer or a source would flag the change, a bounded-change model describes something real. If the subject asserts the value and nothing compares it, it does not.
  • A vendor reports high accuracy under small feature perturbations on a credit model. What do you ask next?
    Which fields were perturbed, who sets each of them, and what a change costs the subject. Moving every column by a small fraction models an adversary that does not exist. What you need is behaviour when one self-declared column is set to a value the applicant chose freely, and how much of the score that single column can move.

saying these in an interview costs you the question

  • Says capping the perturbation budget bounds the applicant
  • Treats every feature as continuously and only slightly movable
  • Assumes a true input exists that the adversary must stay near
  • Quotes small-perturbation robustness as evidence against gaming
  • Calls a large declared change implausible without any check enforcing it

context