Negative transfer can come from a domain gap or a source-task mismatch — how do you tell which?

## Same symptom, two diseases When a pretrained model underperforms a from-scratch control on the target task, the metric tells you *that* transfer failed, not *why*. There are two distinct causes, and they call for opposite responses, so a senior answer names both and says how to distinguish them. ## Cause one: the domain gap The source and target inputs come from different distributions. The features the backbone computes were tuned for statistics that the target data does not have, so from the earliest layers the model is applying the wrong filters to the wrong kind of signal. A clean text example: a backbone pretrained on English newswire, transferred to clinical shorthand notes. The notes share the alphabet and little else — heavy abbreviation, drug names and dosages, dropped function words, telegraphic fragments that are not sentences, a vocabulary the newswire model has largely never tokenised in context. The pretrained model does not merely lack knowledge; the regularities it internalised (long grammatical sentences, journalistic register, general-news entities) actively mispredict here. Beaten by a small model trained directly on the notes, this is a domain-gap failure. The diagnostic is about the raw inputs, before any model: do target and source inputs share vocabulary, register, channel count, resolution, sensor, sampling rate, class balance, typical scale? Wherever they diverge sharply, you have a gap. It is also the cause that improves with more or closer in-domain data, because the missing thing is *exposure* to the target distribution. ## Cause two: the source-task mismatch Here the inputs may look perfectly compatible; the objective is what betrayed you. Every training objective is a statement about which variation matters and which should be ignored, and a representation that is good at the source task is one that has thrown the ignorable variation away. The sharpest example: a face-recognition backbone reused for facial-expression intensity regression. Both tasks take a cropped face — there is barely any domain gap. But identity training explicitly rewards mapping the same person to the same embedding across smiles, frowns, lighting and pose. Expression *is* the nuisance variable the source loss was built to suppress. Reusing those features for expression intensity means starting from a representation that is, by construction, least informative about the target label. Extra target images do not help, because the problem is not exposure — it is that the inherited feature space actively discards the target signal. The diagnostic here is a single question: was the source objective invariant, explicitly or implicitly, to the thing my target label measures? If the answer is yes, expect trouble no matter how similar the pictures look. ## Can fine-tuning rescue a mismatch? Partly, and this is worth stating carefully. The invariance is a learned property of the weights, not an architectural erasure, and the information about expression is still present in the raw pixels and in the earliest layers. Full fine-tuning can therefore claw some of it back. But you are starting in a region of parameter space that was optimised to be uninformative about your label, and that is a worse starting point than random for the later layers. Which is exactly why the from-scratch control so often wins in this case. ## Mixed cases and how to act on them Real failures are often both at once, and that is fine — the diagnosis is about which lever to pull first: - **Gap dominant.** Find a source pretrained on data closer to the target, or run further pretraining on in-domain unlabelled data before touching the target labels. If neither exists and target labels are plentiful, from scratch is a legitimate answer. - **Mismatch dominant.** Change the source task. A backbone pretrained on a generic objective, or on one that preserves the variation you care about, will beat a specialised backbone whose specialisation is the wrong one. A more specific, better-known source is not automatically a better source. - **Both.** Do not spend weeks on transfer. Establish the from-scratch number early and let it set the bar. ## What a weak answer sounds like "The source dataset was too small" or "the model was not big enough" — neither is a mechanism. So is "they are both faces, so it should transfer": similarity of inputs is only half the question, and the half that misleads most often. The senior move is to interrogate the objective as carefully as the data.