On a 300-server Linux fleet with confirmed persistence on nine hosts, which do you rebuild rather than clean?

## Why 'rebuild everything' and 'clean everything' are both wrong answers A fleet-wide rebuild is the safest technical position and it is often unaffordable in the window you have; cleaning in place is affordable and, on a host where the adversary had root, close to unfalsifiable. The interviewer is looking for a tiering rule you can defend to the platform owner who has to execute it, not a slogan. ## The rule: rebuild is forced by what you cannot account for Three conditions each force a rebuild independently. **1. Privilege reached.** If the intruder held root — or you cannot prove they did not — then everything on that host that would attest to its own cleanliness is within their reach: the configuration agent, the package database, file metadata, the local logs, the boot chain. A clean sweep result from that host is the host describing itself. Cleaning is only defensible when the access was bounded to an unprivileged service account and you can show it never escalated. **2. Accountability of actions.** Can you enumerate what ran during the dwell window? If the host had process telemetry, shell history you can trust, and audit records covering execution, you can reason about what was done and confirm that the persistence you removed was the whole of it. If a third of the fleet has no agent and audit was never configured for execution, then for those hosts your list of what the adversary did is a list of what you happened to see — which is not the same list. **3. Cost of being wrong versus cost of the rebuild.** On a configuration-managed, stateless fleet, rebuilding is a pipeline run rather than an artisan effort, and it is frequently cheaper than the analyst-days it would take to argue a host clean. The picture inverts for a stateful host — a database primary, a machine holding data nobody else has — where the rebuild is a project, and where the correct move may be to accept a narrower cleaning with heavier monitoring. ## The trap: what does the rebuild pull from? This is the point that separates a good answer from a rehearsed one. A rebuild is only as trustworthy as its sources: the configuration-management repository and its catalogue, the base image, the internal package mirror, and the credentials the build process injects. An intruder who committed a change into the configuration repository has achieved persistence that survives an unlimited number of rebuilds, and a fleet-wide rebuild becomes a fleet-wide redeployment of their access performed by you at speed. So before the first host is rebuilt: review repository history over the intrusion window, check who could commit and whether their credentials were exposed, verify the base image and mirror against known-good state, and rotate the build pipeline's own secrets. ## What a rebuild does not fix Say this explicitly, because it is where over-confident answers fail: - **Credentials.** Keys, tokens and passwords read off the host stay valid after it is rebuilt. Invalidating them is separate work and it is not optional. - **The entry vector.** If the exposed service or stolen key that admitted them is unchanged, the rebuilt host is a fresh target with the same door. - **Their footholds elsewhere.** A rebuild is a statement about one machine, and eradication is a statement about an estate. - **Your evidence.** The rebuild destroys the host's state. Whatever you will need later — to scope the rest of the fleet, or to defend the claim afterwards — has to be collected before the machine goes. ## Handling the hosts with no evidence either way The hard tier is the one that neither alerted nor produced a usable sweep result: no agent, no drift report, no audit records. They are not clean; they are unknown. Sort them by exposure — reachable from the compromised segment, sharing the credential that was stolen, holding the data that matters — and rebuild the reachable and stateless ones because it is cheap. The remainder get compensating controls and a named watch rather than an assumption. ## Making the argument to the platform owner The platform owner is right that rebuilding 300 servers on a maybe is not free, and the useful reply is not louder insistence. It is a tiered list with a reason per tier: these nine are confirmed and non-negotiable; these are stateless and config-managed, so rebuilding is one pipeline run and no argument is needed; these are stateful, so we propose cleaning plus specific monitoring; these have no evidence and here is what we need — telemetry, a maintenance window, or an explicit acceptance that they stay as they are. That turns an unbounded demand into a set of decisions with owners, which is the form in which it actually gets done.