A stakeholder asks for a model that flags 'high performers' but nobody has defined the term — how do you proceed?

## The request is not yet a machine-learning problem Supervised learning needs a target column. An HR performance dataset where 'high performer' has never been agreed does not have one, and no choice of algorithm creates it. So the first phase of this work is definitional. Skipping it is the most common way this kind of project fails: a target gets picked silently — often whichever column was convenient — and every downstream number then quietly means something nobody agreed to. ## Step 1: find candidate proxies that were actually recorded Ask what outcome the organisation already writes down that correlates with what they mean. Typical candidates: promotion within a defined window, retention past a tenure threshold, attainment against a quota, a manager's annual rating. Each is a **proxy** — a recorded quantity standing in for an unrecorded concept. Write the candidates down and state, for each, what a model trained on it would actually learn. That sentence is the deliverable of this step, because it is what stakeholders will otherwise never hear: - Train on manager ratings and the model learns to *imitate managers*, inheriting whatever leniency, recency effects or group bias the ratings carry. - Train on promotion and it learns *who gets promoted*, which mixes performance with visibility, tenure, and headcount available in each department. - Train on retention and it learns *who stayed*, which is partly about the labour market and partly about who was not being managed out. None of these is 'performance'. Choosing one is a decision about what the organisation is willing to optimise, and it belongs to the stakeholder, made explicitly, not to the modeller, made by default. ## Step 2: test whether the concept is learnable at all Before funding anything, run a small agreement study. Hand two experienced reviewers the same fifty employee records with a written definition and have them label independently. Then measure how often they agree. This is the most informative cheap experiment available, because **inter-annotator agreement bounds achievable model performance**. If two humans applying the same definition disagree on 40% of cases, the labels contain that much irreducible disagreement, and a model scored against one reviewer's labels cannot do better than the other reviewer does. Low agreement is not a reason to give up quietly; it is a finding to report, and usually it means the written definition is under-specified. Sharpen the guidelines, rerun the study, and see whether agreement moves. If it does not move, the concept is not a target — it is an opinion. ## Step 3: price the labels A definition nobody will fund is not a definition. Estimate how many labelled cases the problem needs and what each costs in expert time. This is where supervised framing gets expensive: in digital pathology one label can cost a board-certified specialist twenty minutes, which is exactly why an archive of 90,000 slides carries only 400 labelled ones. HR labels are cheaper per unit but scarcer in a different way, because the population is small and each label consumes a senior reviewer's judgment. Put the number in front of the stakeholder alongside the proxy options. Very often the conversation ends with a better-scoped question, because the cost makes people say out loud what they actually wanted. ## Step 4: consider that the honest deliverable may be descriptive If no defensible target exists and none will be funded, the alternative is not to fake one. It is to change what you deliver: a description of how the workforce actually varies along recorded, uncontroversial dimensions, presented as structure rather than as a verdict on individuals. That answers 'what does our workforce look like' honestly, and it makes clear it is not answering 'who is good'. The failure to avoid here is inventing ground truth: grouping employees, declaring the most flattering group to be the high performers, and then training a classifier to reproduce that assignment. The result looks like a supervised model with an impressive score, but the score only says the classifier can reproduce the earlier partition. Circularity of that kind is very hard to spot once it has been through two pipeline stages and one slide deck. ## Step 5: ask whether the decision should be automated at all With people-affecting targets there is a legitimacy question that sits above the modelling one. Who sees the score, what action it triggers, whether the subject can contest it, and what the proxy's known biases would do to particular groups. A senior answer raises this without being asked, because it is the part a stakeholder cannot be expected to raise on their own. ## What a good answer sounds like in the room A strong candidate reframes the request within a minute — 'there is no target yet, so let me tell you what we would have to agree on' — offers two or three concrete recorded proxies with their distortions named, proposes the agreement study as the cheap next experiment, and states plainly that a descriptive deliverable is a legitimate outcome rather than a failure. The same pattern generalises past HR. Whenever an ask arrives with no recorded outcome behind it — a 40,000-hour archive of pump-vibration telemetry where nobody ever tagged which runs preceded a failure is the same situation with different vocabulary — the options are identical: find a recorded proxy, start instrumenting so the outcome exists in future data, or deliver structure instead of prediction.