Your CI runner kills a build step with exec.CommandContext, yet processes it spawned keep running. Why?

## The symptom A CI runner executes user-supplied build steps on a shared host. A step exceeds its deadline, the runner's context fires, `os/exec` kills the command, `cmd.Wait` returns, and the runner reports a clean timeout. Some time later the host is inspected and the picture is different: a test binary from that job is still running, still writing, and still holding the job's workspace open so the cleanup that should have deleted it silently failed. Its parent PID is 1. ## Why signals stop at the child When a Go program launches a process it gets back exactly one PID, and every signalling API in the standard library — `os.Process.Kill`, `os.Process.Signal` — targets that one PID. There is no "and its descendants" variant, because Unix does not have one at the process level. A build step is almost never a single process. `/bin/sh -c "make test"` is a shell that forks a build tool that forks a compiler, and each of those may fork more. Killing the shell removes one node from the middle of that tree. The rest are not notified: a Unix child is not signalled when its parent dies. They are simply re-parented to PID 1 and continue, which is why they show up later with a PPID of 1 and the original job's working directory still open. This is why the diagnosis is a process listing on the host after the runner has exited. A tree view of the survivors — their PIDs, their parent PIDs, and their process group IDs — tells you immediately that they share the old step's group ID while their parent is now init, which is the signature of exactly this bug. ## Process groups: the mechanism that does exist Unix does have a grouping abstraction one level up: the process group. Every process belongs to one, `kill` can address a whole group, and a negative PID is how you say so — `kill(-pgid, sig)` signals every member. Go exposes this in two places. At launch, `cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}` (a Unix-only struct) asks the runtime to put the child into a new process group of its own. The new group's ID is the child's PID, and — importantly — the descendants it forks inherit that group unless they deliberately leave it. At kill time, `syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)` signals every member of that group. Put together with the cancellation fields, a correct runner looks like this: ```go cmd := exec.CommandContext(ctx, "/bin/sh", "-c", step) cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true} cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGTERM) } cmd.WaitDelay = 10 * time.Second ``` The custom `Cancel` is not optional. The default that `exec.CommandContext` installs calls `Kill` on the process, which is exactly the process-only behaviour you are trying to escape. And note the remaining gap: when `Cmd.WaitDelay` expires, the package's own fallback also terminates only the direct child. A runner that must guarantee nothing survives sends a group-wide SIGKILL itself after `Wait` returns. ## What still escapes Process groups are a big improvement and not a guarantee. A descendant can call `setsid` or `setpgid` and leave the group deliberately — which is what daemonising means, and user-supplied build steps do it by accident all the time when they start a background service "to be helpful". A process stopped or in an uninterruptible state may not be reaped promptly. And a step that writes its own supervisor is free to re-launch anything you killed. There are two Unix details worth knowing here. On Linux `syscall.SysProcAttr` also offers `Pdeathsig`, which asks the kernel to signal the child when its parent dies — but it fires on the death of the parent *thread*, which in a multi-threaded Go program is not the same as the process exiting, and it covers only the direct child. It is a useful backstop, not a solution. And `Setpgid` has a side effect: the child is no longer in the runner's group, so it will not receive a terminal-generated interrupt aimed at that group. Whatever the runner did with those signals, it now has to forward deliberately. Windows has no equivalent field. `syscall.SysProcAttr` there is a different struct entirely, and the real answer on that platform is a job object, which the standard library does not wrap. ## The decision behind the code For a platform engineer bounding what user-supplied code may leave behind, the process-group technique is the floor, not the ceiling. It reliably cleans up the well-behaved 95% and turns "processes randomly survive" into "processes survive only when a step actively detached". If the requirement is absolute — nothing from a job outlives the job — the enforcement has to come from a boundary the child cannot step out of, meaning a per-job cgroup or a per-job container that is torn down as a unit. Signals are cooperative at the edges; a resource boundary is not. The practical middle ground most runners land on: launch every step in its own process group, terminate the group with SIGTERM on cancellation, escalate to a group-wide SIGKILL after a bounded delay, then sweep — after `Wait` returns, check whether anything in that group still exists and log it loudly, because a step that survives a group kill is a step that detached, and that is a policy question about what you allow, not a bug in the runner.