skip to content

You rebuilt a Linux server and kept its hostname and IP address. Now `ssh` to it aborts with `WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!` and refuses to connect. Why does a rebuild trigger that, and what is the correct way to clear it?

level: middleimportance: should knowfreq 58%

answer

  1. the client pinned an identity
  2. host keys live in /etc/ssh
  3. rebuild regenerates them
  4. -R removes one entry, hashed or not
  5. verify out of band first

basics

~20 s

Your client pinned the old server's host public key in ~/.ssh/known_hosts, and the rebuild generated fresh host keys in /etc/ssh. Verify the new fingerprint out of band, then drop the stale entry with ssh-keygen -R <host> and reconnect to record the new one.

solid answer

~50 s

On first connection the client stores the server's host public key in `~/.ssh/known_hosts` keyed by hostname and address. A rebuild regenerates `/etc/ssh/ssh_host_*_key` on first boot, so the key presented no longer matches the pinned one — and since that is exactly what a machine-in-the-middle looks like, the client aborts and refuses password authentication so you cannot hand a credential to an impostor. The correct sequence is: confirm the new fingerprint through a channel that is not the suspect connection — the cloud provider's console output, a serial console, or `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` run on the box — then `ssh-keygen -R server` (and `-R` the IP if you connect that way), reconnect, and accept the new key. For fleets, prepopulate a managed `known_hosts` with `ssh-keyscan`, or sign host keys with a CA so clients trust the CA once and rebuilds stop causing churn.

code

bash · 4 lines
bash
ssh-keygen -F server.example.com
ssh-keygen -R server.example.com
ssh-keygen -R 10.0.4.17
ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

go deeper

for a junior

Know that known_hosts records the server's identity from the first connection, that a rebuild changes it, and that the surgical fix is ssh-keygen -R <host> rather than deleting the file.

for a middle

Explain trust-on-first-use, where host keys live on the server, and why the client also refuses password authentication when the offered key does not match the pinned one.

for a senior

Show the out-of-band verification step — console output or the fingerprint read on the box — and be explicit that the legitimate rebuild and a real interception are indistinguishable from the client.

for a principal

Argue the fleet position: constant warnings train people to click through, so eliminate the churn with distributed known_hosts or host certificates rather than accepting disabled verification in pipelines.

## What `known_hosts` is for SSH authenticates in both directions. Your key proves who you are to the server; the server's **host key** proves the server is the machine you meant to reach. On the first connection the client has nothing to compare against, so it shows the fingerprint and asks you to accept it, then writes the key into `~/.ssh/known_hosts` next to the hostname and address you used. Every later connection compares the offered key against that record. This is trust-on-first-use: the first connection is the weak point, and every subsequent one is protected. The host key pair lives on the server in `/etc/ssh/`, typically `ssh_host_ed25519_key` plus `ssh_host_rsa_key` and their `.pub` counterparts. They belong to the machine, not to any user. ## Why a rebuild breaks it Fresh host keys are generated the first time sshd starts on a new installation — by the package install, by `cloud-init`, or by whatever image pipeline you use. Reinstalling, reimaging, restoring from a different snapshot, or replacing an instance behind the same DNS name all produce a machine with the same *name* and a different *identity*. The client sees a mismatch and refuses, printing the offending line number: ``` @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ Offending ECDSA key in /home/alice/.ssh/known_hosts:12 ``` Note what else it does: it disables password authentication for that connection. If you really were talking to an impostor, the dangerous outcome is not the failed session — it is typing your password into it. Refusing to send credentials to an unverified host is the whole point of the check. ## The correct clearing procedure The legitimate case and the attack look identical from the client, so the step you must not skip is verifying the new fingerprint through a channel that does not depend on the connection you distrust: - read it from the instance's console output or serial console, where `cloud-init` prints the host key fingerprints on first boot; - or run `ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub` on the box via that console; - or take it from the config-management record of what the image installed. Then remove the stale entry and reconnect: ``` ssh-keygen -R server.example.com ssh-keygen -R 10.0.4.17 ssh [email protected] ``` `ssh-keygen -R` rewrites `known_hosts` and keeps a `.old` backup. It works even when entries are hashed — on Debian and Ubuntu the file is stored with hashed hostnames by default, so you cannot find the line by grepping for the name, but `ssh-keygen -F server.example.com` will locate it and `-R` will delete it. Remove the address entry too if you sometimes connect by IP, otherwise the warning returns the next time you do. Deleting the whole `known_hosts` file also "works" and is worth calling out as the wrong habit: it throws away every other host's pinned identity, silently downgrading all of them to trust-on-first-use again. ## At fleet scale, stop the churn instead of clearing it If machines are cattle, this warning fires constantly, and a team that sees it every day stops reading it — which is precisely the failure mode an attacker needs. Two durable answers: - **Distribute `known_hosts`.** Collect keys with `ssh-keyscan` at build time and ship a managed system-wide `known_hosts` through config management, so clients never rely on first-use trust. The value depends on `ssh-keyscan` running somewhere you trust, ideally on the build network right after provisioning. - **Sign host keys with a CA.** Issue each host a certificate over its host key, and give clients a single `@cert-authority` line for the CA. Any correctly signed rebuild is then trusted immediately, and the client file stops changing at all. This is the same mechanism used in the other direction for user certificates, and it is the answer that scales. What is *not* an answer is blanket-disabling host verification in automation. It converts every SSH session on the fleet into an unauthenticated channel, and it is usually done in exactly the places — deployment pipelines with privileged credentials — where the consequences are worst.

  • Why does the client also refuse to send your password when the host key changes?
    Because the failure mode being defended against is credential theft by an interposed host. A wrong host key means you cannot prove you are talking to the intended machine, so OpenSSH disables the authentication methods that would hand a reusable secret to whatever is on the other end, and tells you which line in `known_hosts` conflicts.
  • An engineer fixes this by deleting `~/.ssh/known_hosts` entirely. What is wrong with that?
    It discards every pinned host identity, not just the stale one, so the next connection to every server in the file is an unverified first-use again. `ssh-keygen -R <host>` removes exactly the one entry — and it works on hashed files, where grep will not find the line.
  • How would you keep this warning from firing across a fleet that reimages hosts weekly?
    Stop relying on first-use trust. Either collect keys at build time with `ssh-keyscan` and ship a managed `known_hosts` via config management, or sign host keys with a CA and give clients one `@cert-authority` line. The second scales better: rebuilt hosts present a valid certificate and no client file changes at all.

saying these in an interview costs you the question

  • Deletes the whole known_hosts file
  • Disables host key checking in automation
  • Thinks the warning is about the user's key
  • Accepts the new key without verifying anything
  • Believes host keys are stored per user account

context