What mix of instruction pairs, preference pairs and distilled traces belongs in a fine-tuning set?

## The three shapes A fine-tuning dataset is not one kind of thing, and the composition question is a genuine design decision rather than a formatting detail. **Instruction pairs (SFT data).** One input, one target output. This is the workhorse. It teaches the model the shape of the task: what fields to emit, in what order, at what length, in what register, following what procedure. Most projects are entirely this, and most projects should be — form is what fine-tuning teaches most reliably. **Preference pairs.** A prompt with two responses, one marked chosen and one rejected. Direct preference optimisation and its descendants consume this shape directly. Preference data expresses something instruction pairs structurally cannot: a ranking over acceptable outputs. When the property you want is "do not over-reassure an anxious owner, but do not alarm them either", nobody can write the single target sentence — but anybody can look at two responses and say which is better. **Distilled traces.** Triples of input, intermediate reasoning, and output, generated by a stronger teacher model. These supply not just the answer but the path to it, which is expensive to author by hand and which is how much of the recent wave of small strong reasoning models was built. ## Ordering, and why it is not negotiable SFT first, preference second. Preference optimisation moves the model between behaviours it can already produce; it is a poor tool for teaching a behaviour from scratch. If the model does not yet emit the right output shape, both members of every pair are wrong and the comparison signal is wasted. Get form right with instruction pairs, then use preference data on the residue — tone, calibration, refusal boundaries, verbosity, the choices between two defensible answers. ## Volumes There is no universal ratio, and anyone offering one confidently is guessing. What holds: - The SFT stage carries the volume, because it is doing the heavy teaching — hundreds to tens of thousands of rows depending on how broad the behaviour is. - The preference stage is typically much smaller. It is a nudge on an already-competent model, and it is expensive per row because each row needs two generations plus an adjudication. - Distilled traces sit inside the SFT volume rather than beside it; they are instruction pairs with reasoning attached. The honest framing for an interview is that you would size the preference stage by whether SFT alone left a gap you can articulate, not by a target ratio. ## Building preference pairs well - **Sample on-policy.** Generate both candidates from the model you are about to tune, at the settings it will run at. Pairs sampled from some other model teach the ranking of *that* model's output distribution, which is not the distribution you are correcting. - **Judge, then discard ties.** An adjudicator — human or model — picks the winner. Ties and near-ties should be dropped: they contribute noise rather than signal, and a pair whose two members are indistinguishable is training on a coin flip. - **Watch the margin.** Pairs where one side is obviously broken are easy and teach little beyond what SFT already covered. The informative pairs are the ones where both responses are plausible. - **Validate the adjudicator.** The judge's verdicts are the labels, so the usual concern applies: check its agreement with an expert on a sample before you trust thousands of its decisions. ## Distilled traces: what they cost Teacher-generated data is the cheapest way to get process supervision, and it comes with three bills. **Error propagation.** The teacher's mistakes become training targets, and a plausible-sounding wrong reasoning chain is worse than an obviously wrong answer because it survives casual review. Where the final answer is checkable — a computed value, a category from a fixed set, code that either passes tests or does not — keep only traces whose final answer is correct. That rejection-sampling step is the single highest-value filter available on distilled data. Where nothing is verifiable, you are relying on a judge and should size your confidence accordingly. **Style transfer you did not ask for.** A distilled student inherits the teacher's voice, its hedging habits and its formatting tics along with its competence. If your product has a defined voice, that is a conflict to resolve deliberately in the data, not to discover after training. **Terms of use.** Whether the teacher's provider permits using its outputs to train a competing model is a real constraint that varies by provider and changes over time. This is a question to answer before building the pipeline, not after, and "we did not check" is a poor answer at a lead level. ## A fourth shape worth naming For tasks with a programmatically verifiable outcome, reinforcement fine-tuning against a grader needs no labelled outputs at all — you supply prompts and a scoring function, and the training loop generates its own attempts. The dataset question becomes "do I have a reliable grader and a good spread of prompts" rather than "can I write the answers". Recognising when a task falls into that category changes what you spend your curation budget on entirely. ## How to decide, in practice Start with instruction pairs and see how far they go. Add preference pairs only for a named deficiency that you can state as a comparison and cannot state as a target. Use distilled traces where reasoning is the thing you lack and where you can filter on a verifiable outcome. And keep the composition documented, because six months later the question "why does this model hedge like that" is answered by the data mix, not by the training code.