A vendor's model checkpoint is yours to inspect byte for byte — why does that not reach behaviour they trained in?

## What review of a code change actually depends on When a security reviewer approves a source change, two things are quietly doing the work: a **readable artefact** — statements a human can follow — and a **reference** to compare against, namely the code as it stood before. Provenance tells you the change came from the right party; the review reaches behaviour because the object of review is legible and the difference is small and localised. A published checkpoint has neither property, which is why the reviewing habit transfers so badly to borrowed weights. ## The artefact is not legible A checkpoint is a set of numeric arrays. There are no named routines, no branches to read, no comment explaining an odd constant. Behaviour is *distributed*: the mapping from input to output is produced jointly by the parameters, and there is no sub-region that means one thing while the rest means another. A conditional response the publisher fitted into the model — the model behaves normally except under some condition they chose — is carried by the same parameters that produce the ordinary behaviour. Nothing about a parameter's value marks it as belonging to that conditional. This is also why the intuitive fix of looking for anomalous parameters does not survive contact. Deep models have far more capacity than their task requires; there is representational room to spare, so a conditional need not distort the parameter statistics in any way an inspector would flag, and normal training already produces wide, irregular distributions of values. ## There is nothing to diff against The second instinct is comparative: retrain the model yourself, or obtain a clean copy, and diff. Both fail for the same structural reason. - **Retraining is not bit-reproducible.** Nondeterministic parallel reductions, data ordering, hardware and library differences all shift results. Two honest runs of the same recipe differ in essentially every parameter. - **Legitimate change is not sparse.** Any fine-tune the publisher performed — and publishers fine-tune constantly — moves nearly all parameters. A diff against any plausible reference is dense by construction. So the comparison you would want, of the form *these few numbers changed and here is why*, is not available at any price. The output of the diff is uniform difference, and uniform difference localises nothing. ## What that leaves One channel remains: run the model on inputs and observe outputs. This is a real channel — behaviour is genuinely observable — but notice what it changes about the task. Review is bounded work over an artefact you hold. Evaluation is *search* over an input space that is effectively unbounded, and the conditions that matter were chosen by the publisher, not by you. You are looking for a response that occurs on inputs you were never told to try, in a model that is by design accurate everywhere else. That asymmetry — cheap for the publisher to install, expensive for the recipient to find — is what makes inherited weights a distinct problem rather than an instance of ordinary artefact review. ## Consequences worth stating in an interview - **Byte-level access is not the constraint.** Teams sometimes ask for the raw file as though obtaining it settles matters. You may hold every byte and still be unable to say what the model does. - **Format does not decide this either.** A weight file that cannot execute anything on load is a good thing for entirely separate reasons, and it changes nothing about whether the trained function has a conditional in it. - **Openness narrows trust rather than removing it.** Publishing the corpus and the training recipe moves the question from the weights to the data and the run. Both are large, and connecting corpus to checkpoint requires reproducing the training, which is expensive and almost never done by the recipient. That is a real improvement in what could in principle be examined, not a conversion of trust into a check. - **Small models are a partial exception.** For a small model over a handful of features you can sometimes characterise the learned function directly. That exception does not scale, and interviewers like candidates who name it and then say why it does not help for anything with real capacity. The conclusion a strong candidate reaches out loud: with borrowed weights, assurance about behaviour is bought by evaluation and by the trust relationship with the publisher, and inspection of the artefact contributes almost nothing to it.