skip to content

OpenSSH operations

Running and unblocking OpenSSH on real hosts: the sshd service, deploying keys, the file-permission traps, and reading ssh -vvv when a login is refused. The SSH protocol itself is owned by the Protocols domain — this node is the sysadmin's side of it.

on this pageshow

questions

6

You generated an SSH key pair on your laptop with `ssh-keygen`. What does `ssh-copy-id deploy@server` then do, which of the two files it produced ends up on the server, and where exactly does it go?

level: juniorimportance: must knowfreq 68%

answer

  1. two files, only one travels
  2. the .pub half, appended
  3. per account, not per host
  4. append with >>, never >
  5. modes 700 and 600

basics

~20 s

ssh-copy-id logs in with the credentials you already have, then appends your public key file (for example id_ed25519.pub) to the remote account's ~/.ssh/authorized_keys, creating the directory and file with safe permissions. The private key never leaves your machine.

solid answer

~50 s

`ssh-keygen` writes a pair: a private key (`~/.ssh/id_ed25519`) that stays on your machine and must stay readable only by you, and a public key (`~/.ssh/id_ed25519.pub`) that is meant to be handed out. `ssh-copy-id deploy@server` opens an ordinary SSH session using whatever authentication still works — usually a password — and appends the **public** key as one line to `~/.ssh/authorized_keys` in the *remote* account's home directory, creating `~/.ssh` with mode 700 and the file with mode 600 if they do not exist. From then on the server challenges you to prove you hold the matching private key, so no password crosses the wire. Which account matters: the key lands in the home directory of the user you logged in as, so a key copied to `deploy@server` does nothing for `root@server`. Nothing is installed on the client side.

code

bash · 3 lines
bash
ssh-keygen -t ed25519 -C "alice@laptop"
ssh-copy-id -i ~/.ssh/id_ed25519.pub deploy@server
ssh deploy@server 'cat ~/.ssh/authorized_keys'

go deeper

for a junior

Know that the pair is private plus public, that only the .pub line goes into the remote account's ~/.ssh/authorized_keys, and be able to say the manual cat >> equivalent out loud.

for a middle

Be ready to explain that authorised keys are per-account, that the file is one key per line, and how to match a login in the server log to a specific key by its SHA256 fingerprint.

for a senior

Expect the follow-up about the first login that still prompts for a password: name the wrong-account case and the ownership/modes case, and say how you would confirm each from the server side.

for a principal

Own the position that keys are identities: per-person keys, a documented path for removal when someone leaves, and no shared private key material, because attribution and revocation both collapse without it.

## The two files, and which one is a secret `ssh-keygen -t ed25519 -C "alice@laptop"` produces exactly two files in `~/.ssh`: - `id_ed25519` — the **private** key. This is the secret. It stays on the machine you connect *from*. If it has a passphrase, the passphrase protects the file at rest; it is not sent anywhere. - `id_ed25519.pub` — the **public** key. A single line of text, safe to paste into a ticket, a wiki or a config repo. Authentication works by the client proving possession of the private key against the public key the server already holds. The server therefore only ever needs the `.pub` half. Copying the private key to the server is not "how it works" — it is a compromise of the key. ## What ssh-copy-id actually does `ssh-copy-id` is a shell script shipped with OpenSSH, not a protocol feature. It: 1. Works out which public key to install — from `ssh-agent` if one is loaded, otherwise the default identity files, or the one you named with `-i`. 2. Opens a normal SSH connection to the target using authentication that still works today (password, or an already-working key). 3. Creates `~/.ssh` on the remote side if needed, appends the public key as a new line to `~/.ssh/authorized_keys`, and fixes the modes. 4. Prints how many keys it added and tells you to try logging in. It is deliberately append-only: existing keys in the file are left alone, and it tries to skip a key that is already present. `authorized_keys` is a plain text file where **one line is one key**; a normal account can accumulate many, one per laptop or per automation client. ``` ssh-copy-id -i ~/.ssh/id_ed25519.pub deploy@server ssh deploy@server 'wc -l ~/.ssh/authorized_keys' ``` ## Doing it by hand when ssh-copy-id is absent Minimal images and some appliances do not ship the script. The manual equivalent is one pipeline, and getting the directory mode right in the same command is the part people forget: ``` cat ~/.ssh/id_ed25519.pub | ssh deploy@server \ 'install -d -m 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys' ``` Note `>>`, not `>`. A single `>` silently destroys every other key in the file, which on a shared jump host is an outage. ## The account, not the host, is the unit Authorised keys are per-account. `~/.ssh/authorized_keys` for `deploy` is a different file from the one for `root` or for `alice`. "I copied my key to that box" is not a complete statement — you copied it to one account on that box. Interviewers often probe this with a scenario where key login works for one user and fails for another on the same server. ## Verifying, and telling keys apart The trailing comment on a public-key line (`alice@laptop`, whatever you passed to `-C`) is free text and is the usual way humans label keys; it is not authenticated and can be edited, so treat it as a label, not an identity. To match a line in `authorized_keys` against a key you hold, compare fingerprints: ``` ssh-keygen -lf ~/.ssh/id_ed25519.pub ``` That prints the size, the `SHA256:` fingerprint and the comment. The server logs the same fingerprint when it accepts a login, which is how you find out *which* of five keys a given session used. ## The failure right after this step The classic sequence is: `ssh-copy-id` reports success, and the next login still asks for a password. Two causes dominate. First, the key went to a different account than the one you are now logging in as. Second, the permissions on the remote home directory, `~/.ssh` or `authorized_keys` are too loose, and the server ignores the file — sshd runs with `StrictModes yes` by default and refuses to trust a key file that other users could edit. `ssh-copy-id` sets safe modes on what it creates, but it cannot fix a home directory that was already group-writable. Finally, a key that is in place but never offered is a *client-side* problem: with several keys present, the client offers them in order and a server may cut the attempt off after too many failures, which is why naming the identity explicitly during triage is useful.

  • What does the `-i` flag change about which key `ssh-copy-id` installs?
    `-i` names the identity to install. Point it at a `.pub` file and that key is used; point it at a private key and the matching `.pub` beside it is used. Without `-i`, `ssh-copy-id` prefers keys currently loaded in `ssh-agent` and falls back to the default identity files in `~/.ssh`. Being explicit avoids installing a stale agent key by accident.
  • An account's `authorized_keys` holds five keys. How do you find out which one a particular login used?
    Read the server's auth log. sshd records an accepted public-key login with the key type and its `SHA256:` fingerprint — `journalctl -u sshd` on RHEL, or the auth log on Debian. Then run `ssh-keygen -lf` over the candidate public keys and match fingerprints. The trailing comment on the line is only a human label and can be wrong.
  • Someone proposes copying the same private key to every engineer's laptop so "everyone can get in". What is wrong with that?
    A shared private key destroys attribution and revocation. Logs show one fingerprint for everyone, so you cannot say who ran a command, and when one person leaves you must rotate the key on every server and every laptop at once. Per-person keys make removal a one-line change in one account's `authorized_keys`.

The public key is a lock you mail to the server and it bolts to its door; the private key is the only key that opens that lock, and you never mail a key.

saying these in an interview costs you the question

  • Says ssh-copy-id uploads the private key
  • Thinks ssh-copy-id generates the key pair
  • Overwrites authorized_keys with > instead of appending
  • Believes authorized_keys stores a hash, not the key
  • Assumes a key copied to one account works for root too

context

open as a page

Key-based SSH login to a Linux server fails with `Permission denied (publickey)` even though the right public key is present in `/home/deploy/.ssh/authorized_keys`, and the server's auth log shows `Authentication refused: bad ownership or modes for directory /home/deploy/.ssh`. What is sshd checking, and which ownership and permissions does it require?

level: middleimportance: must knowfreq 62%

basics

~20 s

OpenSSH's sshd runs with StrictModes yes by default: it ignores a key file that anyone but the owner could modify. The account's home directory must not be group- or world-writable, ~/.ssh should be 700 and authorized_keys 600, all owned by that user.

open as a page

You rebuilt a Linux server and kept its hostname and IP address. Now `ssh` to it aborts with `WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!` and refuses to connect. Why does a rebuild trigger that, and what is the correct way to clear it?

level: middleimportance: should knowfreq 58%

basics

~20 s

Your client pinned the old server's host public key in ~/.ssh/known_hosts, and the rebuild generated fresh host keys in /etc/ssh. Verify the new fingerprint out of band, then drop the stale entry with ssh-keygen -R <host> and reconnect to record the new one.

open as a page

You have edited `/etc/ssh/sshd_config` on a remote Linux server you can only reach over SSH. What do you do before and after restarting sshd so a mistake in that file does not lock you out?

level: seniorimportance: should knowfreq 48%

basics

~20 s

Parse-check the file first with sshd -t, keep your current session open, then reload or restart the unit and prove the change from a brand-new second session before closing the first. Have console access or a timed automatic rollback as the backup.

open as a page

A user reports `Permission denied (publickey)` connecting to a Linux server. Walk through what `ssh -vvv deploy@server` tells you about where the failure lies — and what that output fundamentally cannot tell you.

level: seniorimportance: should knowfreq 44%

basics

~20 s

Verbose client output shows which identities the client offered and which methods the server will accept, so it separates a client that never sent your key from a server that rejected it. It cannot show why the server refused — that reason exists only in the server's auth log.

open as a page

You inherit SSH access for a few hundred Linux servers where each account's `authorized_keys` file is maintained by hand. What actually breaks at that scale, and how would you choose between config-management-pushed key files, an `AuthorizedKeysCommand` lookup, and SSH certificates?

level: principalimportance: nice to knowfreq 32%

basics

~20 s

Revocation and inventory break first: a departing engineer's key survives on whichever hosts were missed, and nobody can answer who can reach what. The three options trade convergence speed against a runtime dependency, and short-lived certificates remove per-host key files entirely.

open as a page