skip to content

When may a team let an LLM prompt carry a consequential decision like loan pre-screening at all, and what must it measure before and after launch?

level: principalimportance: nice to knowfreq 14%

answer

  1. decide the use before the wording
  2. a passing score is not sufficient grounds
  3. advisory review can fail through deference
  4. what leaves the prompt
  5. measure at launch, then keep measuring

basics

~20 s

Letting a prompt carry a consequential decision is a product and legal call, not a wording one: passing a bias test is not sufficient grounds. If allowed, measure swap gaps, accuracy and review behaviour continuously, and keep final adverse outcomes out.

solid answer

~50 s

I would start from the use, not the wording. Tamkin et al. (2023) reduced discrimination through prompting and still said good results on their evaluation are not sufficient grounds for high-risk use, and that a human in an advisory role is not by itself a sufficient guardrail. So my default is that a prompt can draft and summarise, while a final adverse outcome about a person stays outside it unless the people accountable, with legal counsel, decide otherwise. If a prompt does contribute, I would strip fields it does not need and put hard rules in code. Before launch I would measure counterfactual gaps with an agreed limit, accuracy against the current process, and abstain rates. After launch I would watch reviewer override rates and appeals by group, and rerun everything on any model change. If the gate fails, the fallback is the manual process.

go deeper

for a junior

Recall that a low bias score does not by itself make a model suitable for decisions about people, and that this choice goes beyond one engineer.

for a middle

Explain which parts move out of the prompt, such as unneeded fields and hard rules, and why swap gaps and accuracy both have to be measured.

for a senior

Lay out the release gate: agreed gap limit, accuracy floor, abstain rates by group, reruns on any change. Add the monitoring that reveals whether review is actually working.

for a principal

Own the go or no-go and its stop rule. Weigh reversibility, contestability and comparators against the full running cost, involve counsel, and accept that declining to automate can be the right result.

## Decide the use before the wording This is a judgment call with no single right answer, and it is a lead's call because the cost of error falls on people outside the team. Everything else on prompt debiasing assumes the prompt should be doing this job. The prior question is whether it should. The authors closest to the evidence are cautious. Tamkin et al. (2023) showed prompting can sharply reduce measured discrimination, and in the same paper: - They do not endorse automated language-model decisions for the high-risk uses they studied, and do not believe doing well on their evaluation is **sufficient grounds** for such use. - They warn that models interact with people through automation bias, so **an advisory human in the loop is not by itself a sufficient guardrail**. - They say discrimination is not the only criterion, and that models must also be evaluated in naturalistic settings on the task itself. - They note such uses are already subject to existing anti-discrimination law, and that governments and societies should influence them rather than single firms alone. Ganguli et al. (2023) likewise do not advocate language models for admissions, and treat that experiment as a proxy. ## A way to sort uses These tiers are one reasonable default, not a standard. Your counsel and risk owners may draw the lines elsewhere. | Use | Example | Default | |---|---|---| | Drafting for a person who owns the text | Job-ad writer | Prompt may carry it; count wording in samples | | Summarising for an independent decider | Digest of an application for an underwriter | Possible with swap tests on the summary and reviewer monitoring | | Ranking or routing people | Ordering a review queue | Treat as a decision: position changes outcomes | | Automatic negative outcome | Rejecting an application with no review | Keep out of the prompt | The questions that move a use up or down: - **Can the affected person notice and contest the outcome?** A silent rejection cannot be appealed. - **Is an error reversible,** and how quickly? - **Is there a comparator?** A prompt can be better than a poor existing process, but you cannot claim this without measuring both. - **Can a rule do it?** Clear eligibility criteria belong in deterministic code, where they can be read and tested. ## What leaves the prompt No wording makes certain things safe, so move them: 1. **Fields the decision does not need.** Remove them before rendering. A published "ignore demographics" instruction asks the model to act as if the profile were redacted; real redaction does not rely on the model's compliance. Proxies remain, which is why the swap test stays, and they are covered with proxy features. 2. **Hard limits and eligibility rules.** Enforce these in code. 3. **The final adverse decision,** under the default above. 4. **Accountability.** A named owner signs the launch. A prompt cannot. ## Measure before launch 1. **Counterfactual swap gaps** on held-out inputs, by group, on names and labels and several paraphrases, against a limit agreed in advance. The limit is a policy choice, and the sign of the gap matters as well as its size. 2. **Task accuracy** against the current process, with a floor, because a prompt can reach a small gap by declining everything or by answering alike for everyone. 3. **Abstain and refusal rates** by group. 4. **Group-level error rates** on real-shaped data. Which criterion applies is covered with fairness criteria, and small slices are noisy. 5. **A check on the eval itself.** Public benchmark results help screen candidate models but do not stand in for this list. Tamkin et al. tested paragraph-long profiles and single attributes, so a longer document or a dialogue needs its own run. ## Measure after launch - **Reviewer override rates by group.** If reviewers almost never disagree, the review may be deference rather than a check. - **Appeals and complaints** by group and reason. - **Reruns on every change** to model, prompt or input format, as a release gate. Small prompt changes can alter outputs, and later models can respond differently. - **Drift in the input population,** since the test set ages. Agree the stop rule in advance: which breach pauses the feature, who can pause it, and the fallback, normally the manual process you must keep working. ## What it costs A team should budget all of this before saying yes: a maintained test set, swap runs on each change (in one scenario, 200 applications x 6 group variants x 3 phrasings is 3,600 calls per run), reviewer time, a logged audit trail with retention rules, and legal review. When that exceeds the value of the automation, the right decision is not to automate this step. That is a good outcome, not a failure of prompting. This is general engineering guidance and not legal advice. Duties vary by jurisdiction and sector, so take specific questions to counsel.

  • Our gap is small and accuracy beats the human process. Can we now remove the human reviewer?
    Those results are necessary, not sufficient. Tamkin et al. say performing well on their evaluation does not justify high-risk use, and your tests cover only the inputs and attributes you chose. Weigh reversibility and whether a person can contest the result, take it to counsel, and consider keeping a person on every adverse outcome.
  • How do you know a human reviewer is a real check rather than a rubber stamp?
    Measure it. Track how often reviewers overturn the model, by group and by the model's confidence, and seed cases where the model's answer is known to be wrong to see whether reviewers catch them. A near-zero override rate is a warning sign, though it needs reading against the model's real error rate.
  • When would you say no outright, even with good test results?
    When people cannot notice or contest the outcome, when errors are hard to reverse, when no comparator exists, or when a jurisdiction restricts the use. Good test results address only what was tested. If the fallback cannot be kept running, that is also a reason to decline, since the team could not pause a failing feature.

  • A low discrimination score means the use is approved
  • A human in the loop makes any LLM decision safe
  • A good enough prompt can make the final call on any consequential decision
  • Testing at launch is enough, as the prompt does not change afterwards
  • An instruction to ignore race does the same job as removing the field
  • Bias testing covers the legal question, so counsel need not be involved