In an insurer's model registry, who should hold promotion authority for a claim-severity model, and what makes that approval a real control?
answer
- the proposer is not the approver
- approve a digest, not a model name
- the card is the basis for refusal
- recorded with identity and time
- reversible approvals can be delegated
basics
~20 sAuthority belongs to whoever is accountable for the outcome, and never to the author of the run. The approval is a control only when it binds one immutable version, is made against a stated basis for refusal, is recorded with identity and time, and is reversible in minutes.
solid answer
~50 sSeparate the roles: the engineer who trained the run proposes, and someone accountable for the claim-severity model's outcome - the owner who carries the pager and the reserving consequence - approves. That approval becomes a genuine control when four things hold. It binds **one immutable version**, identified by its artifact digest, not a model name. It is made against a **written basis for refusal**, normally a model card attached to the version: intended use, out-of-scope use, the evaluation set and metric with results broken out by the populations that matter, known limitations, the input contract, and the rollback plan. It is **recorded on the version** with identity and time. And it is **cheap to reverse**, because an approval that cannot be undone quickly becomes an approval nobody dares give. The two ways it degrades are symmetrical: an approval nobody can fail is theatre, and an approval that takes weeks gets routed around by an emergency path that quietly becomes normal.
go deeper
Understand that putting a model into production is a decision someone is accountable for, and that the person who trained it is not the right person to sign it off alone.
Describe the mechanics: the approval attaches to one immutable version and its evidence, and the platform refuses the stage transition when a required record does not resolve.
Argue from failure modes. Show how you would detect theatre (a refusal rate of zero) and a bottleneck (promotions routed through the exception path), and what evidence the approver must have to refuse credibly.
Own the trade-off between review cost and release velocity: automate the like-for-like refresh, reserve the signature for material change, and justify that split by a rehearsed rollback time rather than by optimism.
## Why this is an authority question, not a tooling question A registry stage transition is a row change. What makes it a **control** is entirely about who may make it, what evidence it requires, and whether the act is recorded and reversible. Any registry can express the transition; very few organisations get the surrounding four properties right, and that is the part an interviewer is probing. For a claim-severity model the stakes are concrete: the model's output feeds reserve amounts, so a systematic over- or under-prediction is a financial statement problem, not only a quality problem. That is why the approver should be the person who owns the consequence rather than the person who wants the model live. ## Separation of duties, stated precisely - The **author** of the run proposes a version for promotion and supplies the evidence. - The **approver** is accountable for the model's behaviour in production and is not the author. - The **platform** enforces that the transition cannot be made by anyone else, and cannot be made at all when required evidence does not resolve. The third point is what stops separation of duties from being a convention. If a determined engineer can set the production stage directly, the rule is a norm; if the transition validates that a run record, an evaluation on the required set, an input contract and an approval identity all resolve, the rule is a mechanism. ## What the approver is actually reading A number cannot be approved or refused. The promotion document - commonly a **model card** attached to the version - states what the version is claimed to be safe for: - intended use, and explicitly **out-of-scope** use (which lines of business, which claim types); - the evaluation set identifier, the metric definition and the value, broken out by the populations that matter rather than as one aggregate; - known limitations and the conditions under which the model is expected to degrade; - the input contract: which feature-definition version and input schema it requires; - the monitoring plan and the rollback plan, including who executes each. The card is what makes refusal possible, because it converts *this model is better* into a set of claims that can be individually checked. ## The two failure modes, and they are symmetrical | failure | how it looks | what it costs | |---|---|---| | theatre | approval always granted, usually same-day, never with a question | the delay of a real control with none of the protection | | bottleneck | approval takes weeks, committee-scheduled, requires a re-presentation | an emergency path is invented, then used routinely | Both end in the same place: promotions that nobody meaningfully reviewed. The measurable tell for the first is a refusal rate of exactly zero over many quarters; for the second it is the share of promotions that went through the exception route. ## The judgment call: how much to automate The defensible position is to make the **cost of the signature proportional to what is new**: 1. A like-for-like quarterly refresh - same inputs, same objective, same population, same decision threshold - clears on the recorded gates and is promoted automatically, with the same rollback guarantee as any other version. 2. A **material change** - a new input, a new population or line of business, a changed objective, a moved decision threshold - requires the accountable owner's signature against the model card. 3. Anything touching the input contract requires it too, because that is what makes a rollback stop being a pointer flip. This is defensible precisely because rollback is cheap. Reversibility is what converts an approval from a gate that must be right into a decision that can be corrected, and a team that has never rehearsed a rollback has no basis for automating anything. ## What good looks like when audited - Every production version carries an approval naming a person, a time and the artifact digest approved. - The approver is demonstrably not the author of the run. - At least some approvals were refused or returned for more evidence, and the reasons are recorded. - The exception path exists, is rare, and every use has a written justification and a follow-up. - The currently promoted version is not assumed to be the newest one, because rollbacks are recorded as events too. None of that requires a particular registry. It requires deciding, in advance, who is accountable and what they are entitled to refuse.
- What document should the approver be reading?A model card attached to that version: intended and out-of-scope use, the evaluation set and metric with results broken out by the populations that matter for claim severity, known limitations, the input contract, and the monitoring and rollback plans. It is the promotion document because it states what the version is claimed to be safe for, and a claim is something you can refuse.
- Does requiring approval conflict with a quarterly retraining cadence?Only if approval reviews everything equally. Scope it to novelty: a like-for-like refresh on the same inputs, objective and threshold clears on the recorded gates, while a new input, population, objective or moved threshold takes a signature. The cost of the ceremony should track what is actually new, not the calendar.
- What makes the registry stage transition itself trustworthy?That it cannot be made by anyone else, cannot be made when required evidence does not resolve, and cannot be made silently. Identity and time on the transition, validation of the run record and input contract as a precondition, and a refusal path that is genuinely used are the three. A stage anyone can set is a label, not a control.
saying these in an interview costs you the question
- Lets the engineer who trained the run approve its own promotion
- Treats the approval as a formality nobody is expected to refuse
- Approves a model name rather than one immutable version
- Records the approval in a chat thread rather than on the version
- Makes approval so slow that the emergency path becomes routine
- Assumes an approval removes the need for a rollback plan