When does a finite-difference gradient check earn a permanent place in a test suite?

## The decision, framed properly The question is not whether finite-difference checking is a good technique -- it is -- but where it belongs in a codebase that a team maintains for years. A permanent test costs runtime on every run, attention every time it goes red, and credibility every time it goes red for a reason nobody can explain. So the standard is: does this test catch a class of bug that nothing else in the pipeline catches, and can it do so without flaking? ## Where it pays **Hand-written derivatives.** A custom loss whose gradient someone worked out on paper, a layer implemented for speed with its own backward rule, a specialised operation with an analytic form. This is code where an algebra slip is genuinely plausible and where no other test would notice: the model still trains, just worse. This is the whole justification for the technique. **Custom operations at their boundaries.** Beyond a happy-path check, probe the shapes and regimes that the derivation treated specially -- an edge case in a padding rule, a branch that triggers only for a particular input size. ## Where it does not pay **Standard compositions.** A model built from primitives whose derivatives are already trusted has no new derivative code, so the check verifies machinery that is verified elsewhere and merely slows the suite down. **Surrogate gradients.** Where a backward rule is deliberately not the true derivative -- a straight-through estimator, for instance -- there is no agreement to expect and the check has nothing to say. **The production model.** Every coordinate costs two forward passes, so checking a large model is arithmetically hopeless. The test model should be a miniature. ## Designing a test that will still be green in a year - **Make it small.** A few channels, a batch of two, a few dozen parameters. The whole check should finish in about a second, which also makes it cheap enough to run on every commit. - **Make it deterministic.** Fixed inputs, fixed initial parameters, every random draw inside the loss pinned to a single value reused across both perturbed evaluations. A test that is stochastic will eventually fail for no reason and then be disabled by someone under deadline pressure. - **Run it in double precision.** In single precision, subtracting two nearly identical losses leaves so few significant digits that the achievable relative error floor sits far above a 1e-7 threshold; a perfectly correct gradient would fail. This is also the diagnosis when a custom geometric or physics loss passes in double and fails in single: the derivation and the backward code are right, and the failure indicts the arithmetic of the check, not the gradient. - **Calibrate the tolerance, do not guess it.** Run the check on a version you are confident about, look at the distribution of relative errors, and set the threshold with margin above the observed worst case. Too tight and the suite becomes flaky; too loose and it stops catching small systematic errors such as a slightly wrong coefficient on a term. - **Choose coordinates deliberately.** With more parameters than you can afford to probe, sample a couple of dozen at random and add fixed picks: at least one bias, one parameter in each distinct branch of the code, one in each tensor the derivation treats differently. Systematic errors are shared across coordinates, so a modest sample finds them with high probability. Keep the sample deterministic in the committed test; randomise it only in a nightly run where a new failure can be investigated rather than blocking a merge. - **Report per-coordinate, not pass or fail.** When it does go red, the distribution of errors across coordinates is what tells you whether you are looking at an algebra bug, a non-smooth operation, or a step-size problem. ## Stating the limit out loud A finite-difference check verifies internal consistency: that the backward code computes the derivative of the forward code that exists. It cannot see whether the forward code computes the objective you meant. A loss that squares the wrong residual, applies a factor in the wrong place, or reduces over the wrong quantity will be differentiated perfectly and the check will pass with room to spare. That is not a weakness of the technique so much as a boundary on what it buys you, and a lead should say it explicitly when the team treats a green gradient check as proof the layer is correct. The complement is a forward-value test: compute the layer's output for a small input by hand or from an independent implementation and assert on the number. The pair -- forward values against an independent reference, gradients against finite differences -- covers both halves. Either alone leaves a hole, and the gradient check is the half people remember to write.