Transfer Learning and Embeddings

The encoder has collapsed: it maps every input to nearly the same point, so all distances are zero and every triplet's loss equals the margin. Hardest-negative mining causes it because the hardest negatives are mostly mislabels and near-duplicates.

A domain gap means the target inputs look statistically unlike the source data. A source-task mismatch means the source objective learned invariance to the very signal the target needs. Ask what the source loss rewarded, and what it deliberately ignored.

Sweep the split point: transfer the first k blocks, retrain the rest, and read target performance against k. Two effects fight — depth makes features source-specific, and an arbitrary split breaks co-adapted layers.

Width sets a capacity ceiling, not quality, while storage and per-comparison cost grow linearly with it. Train at the widest width you can afford, plot the downstream metric against width, and ship the knee of that curve rather than the widest option.

Elastic weight consolidation adds a quadratic penalty pulling every parameter toward its old-task value, scaled by that parameter's Fisher information. Weights the old task depended on become stiff, while unimportant ones stay free to fit the new task.

A domain classifier trained to tell source from target sits on the shared features. The gradient-reversal layer passes activations through unchanged but negates the gradient going back, so the feature extractor learns to defeat it while the task head keeps features useful.

Depth decides how much a layer must change. Bottom blocks hold generic features that transfer almost unchanged and should barely move, while upper blocks are task-specific and the head is random, so rates rise geometrically from bottom to top.

The embeddings occupy a roughly 30-dimensional subspace of a 2048-dimensional space — dimensional collapse. The training objective and data diversity, not the layer width, decided that, so the remaining coordinates cost storage and carry almost no information.

Switch to vectors that represent a word as the sum of its character n-gram vectors. An unseen surname or typo still shares n-grams with trained words, so a vector is composed for it rather than a shared placeholder.

Both need many negatives but buy them differently. SimCLR's negatives are the rest of the batch, so a 4096-sample batch and its memory are the price. MoCo decouples negatives from batch size with a momentum-encoder queue, so modest hardware suffices.

Fix everything except the encoder: one layer and pooling convention, one probe family with an identical tuning budget, identical preprocessing and splits, and a reported label-budget curve. Add a parameter-free nearest-neighbour check as a cross-test, and freeze the protocol before results arrive.

Sweep the distance on trials from speakers held out of training, then pick the point where the false-accept rate matches what the security policy allows and the false-reject rate stays tolerable. The threshold is a policy choice.

When target labels are plentiful and the domain is far from the source. Pretraining supplies a prior worth most when data is scarce; with enough in-domain examples a small model learns better-suited features and costs less to serve.

Pick the source whose labels force the same discriminations your target needs. For near-miss events that means a large human-action video corpus: separating hundreds of action classes demands motion features a still-image corpus never builds.