Running `ssh host /opt/bin/start-agent.sh` never returns to your prompt even though the script finished and printed its last line. The script leaves a background process running. How do inherited file descriptors explain the hang, and what redirection fixes it?

## What ssh is actually waiting for `ssh host command` does not wait for a process — it waits for a stream. The remote `sshd` creates pipes for the command's stdout and stderr and forwards whatever comes out of them to your client. The client stops when those channels close, which happens when the pipe returns end-of-file, which happens only when **every** descriptor referring to the write end has been closed. That is the crux: EOF on a pipe is refcounted, not owned. The remote command exiting closes *its* copies. Any other process still holding a copy keeps the pipe open. ## How the child got a copy When a shell forks a child, the child starts with a duplicate of the parent's entire descriptor table. Descriptor 1 in the child points at the same open pipe as descriptor 1 in the parent. So: ```bash #!/usr/bin/env bash /opt/bin/agent & # inherits fd 0, 1, 2 from the script echo "agent started" # this line reaches you fine ``` The script exits. `sshd` reaps it and knows the exit status. But `agent` is alive, holding write ends of both pipes, so the channels stay open and the client keeps waiting for output that will never come. You see the last line of the script, then nothing, until you press `~.` or the agent dies. The command *succeeded*; the session is what is stuck. ## The fix is a redirection, not a signal Give the child its own descriptors so nothing it holds points back at ssh: ```bash nohup /opt/bin/agent >/var/log/agent.log 2>&1 </dev/null & ``` Each part earns its place: - `>/var/log/agent.log` replaces fd 1 with a file, so the child no longer holds the stdout pipe. - `2>&1` duplicates that file onto fd 2, releasing the stderr pipe as well. (Order matters — after the stdout redirection, as always.) - `</dev/null` replaces fd 0. Without it the child keeps the ssh stdin pipe open, and if it ever reads it can steal input or block. If the agent should be silent rather than logged, `>/dev/null 2>&1` does the same job. ## Descriptors above 2 count too The rule is about *every* descriptor, not just the standard three. A script that opened its own channel earlier hands that to children as well: ```bash exec 3>/var/run/deploy.lock ... /opt/bin/agent 3>&- >/var/log/agent.log 2>&1 </dev/null & ``` `3>&-` closes descriptor 3 for that command only, so the long-lived agent does not sit on a lock file or an open log the script meant to release. Forgetting this is how a deleted log file keeps consuming disk space (`df` shows it used, `du` cannot find it) and how a lock outlives the run that took it — the space and the lock are freed only when the last descriptor closes. ## Related but distinct problems Two failures live near this one and are worth separating cleanly in an interview, because conflating them is the usual mistake: - **Child dies when the session ends.** That is signal delivery on the controlling terminal, and `nohup` or `setsid` addresses it. Nothing to do with pipes. - **Session hangs though the command finished.** That is this problem: open descriptors keeping a stream from reaching EOF. `nohup` alone does *not* fix it, because `nohup` only redirects stdout (and only when it is a terminal) — under ssh the descriptor is a pipe, not a terminal, so `nohup` leaves it attached and the hang remains. Write the redirections explicitly. ## Confirming it rather than guessing On a Linux host, `ls -l /proc/<agent-pid>/fd` shows what the stray process is holding; a `pipe:[12345]` entry on fd 1 is the smoking gun, and `lsof -p <pid>` gives the same answer with names. Once you can see the pipe in the child's table, the diagnosis stops being a theory. ## The durable rule Anything you leave running after the script exits must be handed its own stdin, stdout, stderr and any extra descriptors — closed or pointed somewhere durable. A background process is not detached until its descriptor table says so. In practice this is why a real long-lived service belongs under a supervisor that sets up its descriptors deliberately, rather than being forked from a script over ssh at all.