A signed release jar calls an endpoint that exists in no source file - how do you investigate?

## Frame the problem correctly before touching anything The scenario: a payments SDK published as a jar, consumed by many downstream services that handle card data. Someone finds an outbound call in the shipped bytecode that appears in no source file, no reviewed diff and no dependency's documented behaviour. The release verifies against the publisher's signature. Two framing errors will wreck the investigation if you make them. **"It is signed, so the artifact is fine."** A signature carries two claims: a particular key vouched for these bytes, and the bytes have not changed since. It carries no claim that the bytes correspond to any source revision. Signing is the last step of the build, so injection anywhere earlier is signed along with everything else. A validly signed backdoored artifact is the *expected* output of a compromised builder, not a contradiction. **"It must be in the source somewhere."** Grepping the repository harder is the most common wasted day. If this is build-time injection, the repository is genuinely clean, and every hour spent there is an hour the builder stays live. ## Step one: preserve and pin the evidence Pull the published artifact from wherever consumers get it, record its digest, and keep it. Record which released versions are affected as far back as you can check - a dormant payload may have shipped several releases before anyone noticed the call. Do not delete the release yet; you will need it, and consumers already have copies. ## Step two: establish the delta Build the tagged source yourself, on a machine that is not part of the pipeline, and compare the result with what shipped. You are not trying to produce an identical file; you are trying to *localise*. Which classes exist in the published jar that your build does not produce? Which methods differ? Is the call site in application code, in a bundled dependency, or in generated or repackaged code? The answer tells you which stage of the build introduced it - compilation, post-compile rewriting, dependency bundling, or packaging. This is also the moment to check the boring explanation: a dependency you legitimately bundle may be the thing making the call, in which case this is a compromised-component problem rather than a compromised-builder one, and the response is different. ## Step three: enumerate what the build was allowed to execute List every actor with write access to the output between source and signature: | Where | What to check | | --- | --- | | Build configuration | scripts, custom tasks, anything that downloads and runs | | Build plugins | the plugins themselves *and their transitive dependencies* | | Post-compile steps | annotation processors, bytecode rewriters, shading, obfuscation | | Bundling | which dependency versions actually landed inside the jar | | Environment | the runner image, preinstalled toolchain, injected agents, env vars | | Publishing | the step that handles final bytes and holds the signing key | Then diff that list against the last release you have independent reason to trust. Injection almost always correlates with something that appeared or moved: a plugin version that floated, a base image that was rebuilt, a build-only dependency that gained a new transitive edge. ## Step four: contain on the assumption the builder is owned If code was written into the artifact during the build, something had execution on the builder. Therefore: - **Every secret the build could read is burned.** Registry credentials, cloud tokens, and the signing key material if it lived anywhere the build could reach. Rotate, do not "monitor". - **Do not rebuild and republish from the same machinery.** A fix built by the compromised builder is a compromised fix, and republishing also destroys the state you wanted to examine. - **Consumers matter more than you do here.** You are the upstream; downstream services have the artifact embedded and running next to card data. Their exposure window is your release history, not your incident timeline. ## What you can and cannot conclude, stated honestly An auditor handed the git history of a build they did not run can conclude what the *intended* inputs were and who approved them. They cannot conclude anything about the released binary. To make a claim about the binary you need evidence about the *execution* that produced it - what ran, on what, with what inputs - which is a different kind of claim from what is inside the artifact or who signed it. If your organisation cannot produce that evidence today, the correct answer to "was release 4.2.1 built from tag v4.2.1?" is "we cannot demonstrate that", and saying so is more valuable than a confident guess.