In a rotation or jigsaw pretext, how do you detect that a shortcut solved the task?

## What a shortcut is A pretext task is a proxy: you invent a label from the raw input — which of four rotations was applied, which permutation shuffled the tiles, what belongs in the cut-out region — and hope that solving it forces the encoder to learn something transferable. The hope is only justified if there is **no cheaper way** to produce the answer. A shortcut is a cheaper way: a low-level, local, incidental regularity that predicts the pretext label without requiring any of the structure you were after. Networks find them reliably, because gradient descent has no preference for the solution you had in mind. Three canonical ones on this leaf: **The watermark rotation shortcut.** You train a network to predict which of four rotations was applied to a product photo. The catalogue images all carry an upright logo watermark in a corner. The rotation of the watermark is a perfect, trivially detectable label. The network reads a few characteristic strokes, predicts the rotation almost perfectly, and never forms any notion of which way up a chair or a shoe belongs. **Chromatic aberration and border continuity in jigsaw.** You cut an image into tiles, shuffle them, and ask the network to name the permutation. Two leaks exist. Lens chromatic aberration makes colour fringing vary systematically with distance and direction from the optical centre, so each tile carries a faint signature of where in the original frame it came from — enough to reconstruct the arrangement without ever looking at objects. Separately, if tiles are cut adjacently, textures and edges continue exactly across their shared borders, so matching pixel rows along the seams solves the puzzle as a jigsaw of paper, not of content. **Texture copying in inpainting.** You blank a region and ask the network to fill it. A plausible-looking fill can be produced by extending the surrounding texture inwards. The reconstruction loss falls, the outputs look convincing at a glance, and the encoder has learned a texture continuation operator. ## How you detect one **Symptom first.** The pattern to watch for is *pretext success decoupled from downstream benefit*. Pretext accuracy that rises very steeply in the first epochs and saturates near ceiling is suspicious on its own — a task that genuinely requires semantics should be hard for a while. The decisive evidence is the comparison you must run anyway: fine-tune the pretrained encoder on your real labelled task and compare against the same architecture trained from scratch and against any other pretrained starting point you have. A pretext solved by a shortcut typically gives you nothing, or nearly nothing, over random initialisation. **Then ablate the cue.** This is the part candidates usually miss. Form a hypothesis about which regularity is doing the work and destroy it in the input: - Suspect the watermark? Crop or blank that corner, or randomise the watermark's own orientation independently of the image, and re-measure pretext accuracy. If accuracy holds up, the watermark was not the cue; if it collapses towards chance, it was. - Suspect chromatic aberration? Convert to grayscale, or randomly drop or shuffle colour channels, and retrain the pretext. A large drop implicates the colour fringing. - Suspect border continuity? Sample tiles with random gaps between them and jitter each tile within its cell so seams no longer line up. This is exactly why gaps and jitter became standard practice in tile-based pretexts. - Suspect texture copying in inpainting? Look at the filled regions. A texture-copy solution is locally plausible and globally wrong: it continues the background straight through where an object should be. Also try holes placed on object centres rather than uniformly at random. **Look at errors and at attribution.** Where does the pretext fail? If rotation accuracy is near-perfect on watermarked catalogue images and near chance on the un-watermarked subset, you have your answer without any further experiment. Saliency over the pretext head pointing consistently at a fixed corner of the frame, rather than at the object, is corroborating evidence. ## How you fix one The correct response is to remove the cue, not to escalate the difficulty blindly. Blank or randomise the watermark; jitter and gap the tiles; drop colour where colour is the leak. Escalating difficulty without removing the leak just makes the shortcut harder to exploit while leaving it the cheapest path. The deeper judgement is that hand-designed pretexts carry this risk structurally. Every artificial task you invent brings its own set of incidental correlations with the label you synthesised, and you cannot enumerate them in advance. That is a reason to design pretexts whose label depends on content that cannot be read off locally, and a reason to treat downstream transfer as the only trustworthy scoreboard: pretext accuracy is a training diagnostic, never a measure of representation quality.