A vendor's robustness table states its norm and radius -- what must you still decide before accepting it?
answer
- complete is not the same as relevant
- whose attacker does that ball describe
- standard deviations of whose distribution
- error under attack is not settlement
- three calls: accept, re-state, decline
basics
~20 sDecide whether the ball they chose describes any attacker your deployment faces. A fully stated budget is still their assumption about who the adversary is, in their units, over coordinates your attacker may not be able to write.
solid answer
~40 sA complete budget is a complete description of a hypothetical attacker, and adopting it adopts their assumption about yours. Three things stay yours. Coordinate scope: their ball let every input value move, while a participant in your market writes only their own postings. Units: a radius over standardized inputs is a count of standard deviations nobody converted back into your quantities and prices, so you cannot say what it permits. Harm: forecast error inside a ball is not the money a moved forecast settles. You can accept the number while recording that it is not a claim about your system, require the budget re-stated in your scope and units, or decide this layer faces no adaptive adversary and spend elsewhere. What you cannot do is read a printed radius as a grade.
code
text · 12 linesRobustness summary (supplier deck, demand-and-price forecaster)
threat model radius attacker access worst-case MAPE
per-coordinate, standardized 0.02 weights, gradients 6.1% (clean 4.8%)
per-coordinate, standardized 0.05 weights, gradients 9.7%
total-energy, whole window 1.00 weights, gradients 8.4%
...
reader's notes:
- radius is in standardized units; deck never converts 0.02 into energy units or price
- the ball spans every window coordinate; one participant writes only their own postings
- metric is forecast error, not the capacity committed or the money settledgo deeper
Know that a supplier's robustness number describes the attacker they assumed, and that assuming the same attacker is a choice somebody makes.
Be able to point at the three gaps in such a table: which coordinates were allowed to move, what units the radius is in, and what the metric measures.
Demonstrate you would convert the radius into deployment units and the metric into the decision it drives before treating the claim as evidence of anything.
Own the call among accepting with recorded caveats, funding a re-statement in your scope and units, or declaring the layer out of scope -- and be able to defend it to a buyer who wants a single grade.
## The situation You are reviewing a supplier's model in a procurement deck. The robustness table is fully specified -- norm named, radius given, attacker access stated, a metric per row. Nothing is hidden. The question in front of you is not whether the table is complete; it is what the completed table is *worth as a claim about your deployment*, and that is a judgment somebody has to own and could refuse. ## Their ball is their assumption about your attacker A norm and a radius describe an attacker. If you accept the table as a robustness rating, you have adopted the supplier's description of who is attacking you -- a description they wrote without knowing your venue, your participants or your settlement. Three gaps usually sit between their attacker and yours. **Coordinate scope.** Their ball almost certainly allows every input coordinate to move. In a marketplace forecaster the input window is supplied by many parties, and any one attacker writes only their own postings. The modelled attacker is stronger than yours in reach, which is not conservatism you can bank -- it is a different exercise, and it may also hide the block that matters by averaging it into coordinates nobody can touch. **Units.** A radius on standardized inputs is a number of standard deviations. Standard deviations of *their* training distribution, over coordinates carrying your energy quantities and your prices. Until somebody converts that number back into units a trader would recognise, no one in the room can say whether the permitted change is trivial or large, and the discussion drifts into whether the number "sounds small". **Harm.** The metric is forecast error under attack. The thing your organisation cares about is the decision the forecast drove -- capacity committed, positions taken, money settled. A modest degradation in error can be a large number in settlement, or none at all, and the table does not distinguish those cases. ## The decisions actually available **Accept, and write down what you accepted.** Legitimate. Record that the claim covers an attacker with write access to the entire window at a radius stated in standardized units, that no mapping to a participant's real capability exists, and who owns re-testing and when. This is honest and cheap, and it stops the number being quoted next quarter as though it meant something else. **Require a re-statement, and fund it.** Ask for the budget expressed over the coordinates a single participant writes, with the radius converted into your units, and the harm measured at the decision. This is real work for somebody, and the reviewer's job includes saying who pays -- the supplier as a condition, or your own red team as a line item. Note that asking for a *larger* radius is the wrong ask: that answers a stronger version of their attacker, not yours. **Decline the layer.** Sometimes the correct call is that this deployment does not face an adaptive adversary at the input layer at all -- participants are identified, postings are logged, and the fraud and market-abuse controls that already exist are the effective defence. Then the assurance budget buys more elsewhere, and you say so explicitly rather than letting an unexamined table stand as coverage. What is not available is treating a printed radius as a grade. A number with its threat model fully stated is still a number about somebody else's threat model. ## Why this is a judgment and not a calculation There is no rule that converts a supplier's ball into a verdict, because the missing information is about your deployment and cannot be recovered from their table at any effort. So the decision rests on how much the model's failure would cost, whether an attacker with the required standing exists in your venue, and what you can afford to verify. Different reviewers can reach different answers on the same table and both be defensible -- what is not defensible is not making the call and letting the table pass as though it had. ## The line to hold Say what the claim covers, in your own units, in one sentence. If you cannot write that sentence, you have not evaluated the claim; you have filed it.
- The supplier offers to rerun the evaluation at a larger radius. Is that what you asked for?No. A larger radius over the same coordinates answers a stronger version of their attacker, not yours. The change that matters is scope and units: bound the coordinates one participant actually writes, express the radius in your quantities and prices, and report the harm at the decision. A bigger number in the wrong units is still uninterpretable.
- You accept the model despite the gap. What do you write down?That the claim covers an attacker permitted to move every input coordinate at a radius stated in standardized units; that nobody has mapped that radius to what a participant can legally post; that harm was measured as forecast error rather than at settlement; and who owns re-testing, on what trigger, by when. The point is that the next reader inherits the caveats, not just the number.
saying these in an interview costs you the question
- Reads a printed radius as a robustness grade
- Adopts the supplier's ball as the deployment's threat model
- Accepts a standardized radius nobody converted into real units
- Asks for a larger radius instead of the right coordinate scope
- Files the table as coverage without recording what it covers