A long-running command you started in an interactive SSH session keeps dying whenever the connection drops. Which signal kills it, what makes the kernel send that signal, and how would you start the job so it survives?
answer
- nothing crashed — it was signalled
- the terminal, not sshd, is the origin
- session leader and controlling terminal
- one tool ignores it, one escapes it
- ignored dispositions survive exec
basics
~20 sSIGHUP kills it. When the SSH connection drops the kernel hangs up the controlling terminal and signals the session, and SIGHUP's default action is to terminate. Start the job under nohup or setsid, or inside a terminal multiplexer.
solid answer
~50 sIt is SIGHUP. Your shell is the leader of a session whose controlling terminal is the pseudo-terminal `sshd` created; when the connection drops that terminal is hung up, the kernel sends SIGHUP to the session leader, and the shell relays it to the jobs it owns. SIGHUP's default action is terminate, so anything with no handler dies. There are two ways out. Make the job ignore the signal — that is exactly what `nohup` does before it execs the command, and an ignored disposition survives `exec`. Or take the job out of the session so no hangup can reach it: `setsid`, or a `tmux`/`screen` session that keeps running on the server. Either way redirect stdin and stdout, because the terminal they pointed at is gone. Daemons have no controlling terminal, which is why they are free to reuse SIGHUP to mean "reload configuration".
code
bash · 2 linesnohup ./import.sh > import.log 2>&1 & # ignores SIGHUP, same session
setsid ./import.sh > import.log 2>&1 < /dev/null & # new session, no terminalgo deeper
Be able to name SIGHUP as the signal that reaches jobs when a terminal session ends, and know that nohup ./job & or a tmux session is how you keep long work alive across a disconnect.
Explain the chain: pseudo-terminal, session leader, hangup, default terminate action. Say clearly which tool changes the signal disposition and which one changes the session, and mention that output must be redirected because the terminal is gone.
Show judgment about which mechanism belongs where — an ad-hoc nohup for a one-off, a multiplexer when you need to reattach, and a proper supervised service for anything that matters. Be ready to explain why a job that must survive an operator's laptop should not have been started from a login shell at all.
Own the standard: long-running work should live under a supervisor with a defined restart and shutdown policy, not under an engineer's SSH session. Be able to argue for making these jobs restartable and idempotent so a lost connection is an inconvenience rather than a data-integrity question.
## The symptom You start a build, an import or a long `rsync` over SSH, your laptop sleeps or the Wi-Fi flaps, and when you reconnect the work is gone with no error of its own. Nothing crashed. The job was signalled. ## Where the signal comes from When you log in over SSH, `sshd` allocates a pseudo-terminal and starts your shell on it. That shell becomes a **session leader**, and the pseudo-terminal becomes the session's **controlling terminal**. Commands you run become process groups within that session, one of which is the terminal's foreground process group at any moment. When the connection drops, the far end of that pseudo-terminal is closed — a *hangup*. The kernel's response is defined: it sends SIGHUP to the controlling process of the session, which is your shell. An interactive shell that receives SIGHUP normally relays it to the jobs it still owns before exiting. Either way, the signal lands on your long-running command. SIGHUP's default action is to terminate the process. Your job installed no handler for it, so it dies quietly — no core dump, no message of its own, which is why nothing in its log explains anything. ## The two escapes, and how they differ There are exactly two mechanisms, and interviewers like this question because candidates often conflate them. **Ignore the signal.** `nohup` sets SIGHUP's disposition to `SIG_IGN` and then execs your command. This works because of a rule in the signal model: across `exec`, handlers are reset to the default, but a signal that is *ignored* stays ignored. So the new program starts life immune to SIGHUP without knowing anything about it. The job stays in the same session — it is simply deaf to the hangup. ```bash nohup ./import.sh > import.log 2>&1 & ``` `nohup` also redirects standard output to a file named `nohup.out` if you have not redirected it yourself, precisely because the terminal is about to vanish. **Leave the session.** `setsid` runs the command in a brand-new session with no controlling terminal at all. There is now no terminal to hang up, so the kernel never generates SIGHUP for it. This is the same move a classic daemon makes when it detaches from its launching shell. A terminal multiplexer such as `tmux` or `screen` is the third practical answer and is really a variant of the second: the multiplexer server keeps running on the host with its own pseudo-terminals, and your SSH client is merely attached to it. When the connection drops, the client goes away and the server — with your shell and jobs inside it — does not. This is the option to reach for when you want to *come back* to an interactive session rather than just survive. ## The trap after you survive Surviving the hangup is not the same as still working. Your job's standard output and standard error may still point at the pseudo-terminal that no longer exists; writes to a hung-up terminal fail with `EIO`, and a program that treats a failed write as fatal will die anyway a moment later. That is why every one of these recipes redirects output to a file and redirects stdin from `/dev/null`. If your job asks for input after the terminal is gone, it gets an error, not a prompt. ## Why daemons reuse SIGHUP A daemon has deliberately detached from any controlling terminal, so it can never receive a genuine hangup — the signal is unused for it. That freed it up for a long-standing convention: SIGHUP means "re-read your configuration and reopen your log files without restarting". This is why log-rotation tooling signals daemons rather than restarting them, and it is why a daemon must actually install a handler for SIGHUP: if it does not, the default action still applies and a well-meant reload becomes an outage. ## What an interviewer is checking The named signal is the easy half. The half that separates people is knowing *who* generates it and *why* — that it comes from the terminal layer and the session, not from `sshd` deciding to be tidy — and being able to say which of `nohup` and `setsid` changes the disposition and which changes the session. Getting that pair the wrong way round is the standard tell.
- How does `setsid` differ from `nohup` in what it actually changes?`nohup` changes the signal disposition: it sets SIGHUP to ignored before exec'ing your command, so the job stays inside the session but no longer reacts to the hangup. `setsid` changes the process's session: it starts the command in a new session with no controlling terminal, so the kernel never has a reason to send SIGHUP to it at all. One is deafness, the other is absence.
- Why do so many daemons treat SIGHUP as "reload configuration"?Because a daemon detaches from its controlling terminal, so it can never receive a real hangup — the signal is spare. The convention grew up around that, and it is why log rotation signals a daemon instead of restarting it. The important caveat is that the daemon must install a handler; with the default disposition still in place, a reload request terminates it.
- You reconnect and find the job survived but produced no output. What happened?Its standard output still referred to the pseudo-terminal that was destroyed with the session. Writes to a hung-up terminal fail with `EIO`, so the output went nowhere and a program that treats write failures as fatal may have exited shortly after surviving the signal. That is why these recipes always redirect output to a file and stdin from `/dev/null`.
saying these in an interview costs you the question
- Says sshd kills the job when the client disconnects
- Thinks nohup detaches the job from the session
- Believes tmux and nohup solve the problem identically
- Assumes SIGHUP always means reload, never terminate
- Forgets output still points at a destroyed terminal