skip to content

Service Lifecycle

5 roadmaps6 questions

Driving a service with systemctl start, stop, restart, reload, enable, disable and mask; reading its state (active, inactive, failed, activating); and controlling ExecStart, ExecReload and ExecStop plus the Restart= policy. Interviewers ask because enable-versus-start and a restart loop quietly masking a crashing process are everyday operational traps.

on this pageshow

explore

questions

6

On a systemd host, what is the difference between `systemctl reload nginx` and `systemctl restart nginx`, and why does `systemctl reload` fail outright on some units?

level: juniorimportance: must knowfreq 68%

answer

  1. same process, or a fresh one
  2. one is two jobs stop then start
  3. only one directive is executed
  4. no ExecReload means no reload
  5. daemon-reload is a third, different verb

basics

~20 s

systemctl restart stops the service and starts a new process, so the PID changes and in-flight work is dropped. systemctl reload runs only the unit's ExecReload= command, leaving the same process running; a unit that defines no ExecReload= rejects reload with an error.

solid answer

~40 s

`systemctl restart` is really two jobs: a stop job then a start job. The old process is signalled, terminated and replaced, so the main PID changes, open connections die and any in-memory state is lost. `systemctl reload` runs a single directive from the unit file — `ExecReload=` — which is normally something like `/bin/kill -HUP $MAINPID` or the daemon's own `-s reload` command. The process survives, keeps its PID and its listening sockets, and simply re-reads its own configuration. If the unit file has no `ExecReload=`, systemd has nothing to run and refuses the job with "Job type reload is not applicable for unit ..." — which is why `systemctl reload-or-restart` exists for scripts that must not care. Note that neither verb re-reads the *unit file*; that is `systemctl daemon-reload`.

code

ini · 9 lines
ini
[Unit]
Description=Example API

[Service]
ExecStart=/usr/local/bin/exampled --config /etc/exampled.conf
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

go deeper

for a junior

Know that restart replaces the process and reload asks the running process to re-read its config, and be able to say that not every unit supports reload.

for a middle

Explain that reload executes the unit's ExecReload= directive and nothing else, and separate it cleanly from daemon-reload, which re-parses unit files into systemd's own memory.

for a senior

Show judgment about when a reload is insufficient — new binary after an upgrade, changed Environment= or limits, a changed listen address — and use reload-or-restart in automation that cannot know.

for a principal

Own the operational contract: which services must be reloadable for config rollouts, whether a reload is genuinely connection-preserving in each daemon, and how deploy tooling should choose between reload, try-restart and a full restart.

## Two verbs, two kinds of job systemd models every operation as a *job* on a unit. `systemctl restart foo.service` enqueues a stop job and then a start job for the same unit. The running process is signalled and reaped, and a brand-new process is spawned from `ExecStart=`. Everything that lived in that process is gone: its memory, its accepted connections, its file descriptors, its uptime. `systemctl reload foo.service` enqueues a *reload* job. A reload job has exactly one implementation — the command line in the unit's `ExecReload=` directive. systemd runs that command, waits for it to finish, and the unit stays in the state it was already in. The service process is never stopped. ```ini [Service] ExecStart=/usr/sbin/nginx -g 'daemon off; master_process on;' ExecReload=/usr/sbin/nginx -s reload ``` A very common form uses systemd's `$MAINPID` specifier, which expands to the PID systemd is tracking as the unit's main process: ```ini ExecReload=/bin/kill -HUP $MAINPID ``` ## Why some units simply refuse to reload Reload is not a generic capability that systemd can synthesise. If the daemon has no way to re-read its configuration in place, the packager leaves `ExecReload=` out, and then: ``` $ systemctl reload foo.service Failed to reload foo.service: Job type reload is not applicable for unit foo.service. ``` That is not a bug and not a permissions problem — the unit genuinely does not support the operation. `systemctl reload-or-restart foo.service` is the portable verb: it reloads when the unit supports it and restarts otherwise, which is what post-install scripts and configuration-management tools use. ## The three-way confusion: reload, restart, daemon-reload These three are asked about together because they are constantly mixed up. - `systemctl reload foo` — tells the *service* to re-read *its own* config (`/etc/nginx/nginx.conf`, and so on). - `systemctl restart foo` — replaces the service process. - `systemctl daemon-reload` — tells *systemd itself* to re-read *unit files* from disk and rebuild its dependency graph. Editing `/etc/systemd/system/foo.service` and then running `systemctl restart foo` does **not** pick up your edit: systemd starts the unit from its in-memory copy. Modern systemd notices and warns: ``` Warning: The unit file, source configuration file or drop-ins of foo.service changed on disk. Run 'systemctl daemon-reload' to reload units. ``` The correct sequence after editing a unit file is `systemctl daemon-reload` and then `systemctl restart foo`. ## What reload can and cannot do Reload is limited by what the daemon implements. Typical daemons will pick up new configuration files, re-open log files, and re-read TLS certificates. Typical daemons will **not** pick up a new binary after a package upgrade, a changed `Environment=` or `EnvironmentFile=` in the unit, new resource limits, or a changed listening address on an already-bound socket. All of those need a restart, because they are properties of the process systemd spawned, not of the config the process reads. Reload is also not a guarantee of zero dropped requests. Whether a reload is graceful depends entirely on the daemon: nginx forks new workers and drains the old ones, so it genuinely is; a small service that just re-parses its config file and keeps serving is too; something that closes and re-opens its listener during reload is not. ## The other lifecycle verbs worth knowing - `systemctl try-restart foo` — restart only if the unit is currently running; do nothing if it is inactive. Packaging scripts use this so that upgrading a package does not *start* a service the operator deliberately left stopped. - `systemctl reload-or-try-restart foo` — the combination of the two. ## Reading the result `systemctl status foo.service` is the confirmation. After a reload, `Main PID:` and the `Active: active (running) since ...` timestamp are unchanged — proof that the process survived. After a restart, both change. If someone claims they reloaded but the PID moved, they restarted.

  • You edited a unit file under /etc/systemd/system and ran systemctl restart — why did nothing change?
    systemd serves units from an in-memory copy of the unit files. A restart re-runs `ExecStart=` from that cached definition, so your edit is ignored until `systemctl daemon-reload` re-parses the files and rebuilds the dependency graph. Recent versions print a warning about the unit file having changed on disk. The correct order is `systemctl daemon-reload` followed by `systemctl restart <unit>`.
  • Why would a package's post-install script use try-restart rather than restart?
    `try-restart` restarts the unit only if it is currently running, and does nothing if it is inactive. That means upgrading a package never starts a service the operator had deliberately stopped, while still replacing the process for anyone actually running it. `reload-or-try-restart` adds the preference for a reload where the unit supports one.
  • After a reload, how do you prove the service process really survived?
    Compare `systemctl status` before and after: `Main PID` and the `Active: active (running) since ...` timestamp stay identical across a genuine reload, because no new process was spawned. If either value changed, what actually happened was a restart. `systemctl show -p MainPID <unit>` gives the same evidence in a scriptable form.

saying these in an interview costs you the question

  • Thinks systemctl reload re-reads the unit file
  • Assumes every service supports reload
  • Says restart keeps the same PID
  • Claims reload is always zero-downtime regardless of the daemon
  • Confuses daemon-reload with restarting all services

context

resolving links…

open as a page

In a systemd service unit, what is the difference between `Restart=on-failure` and `Restart=always`, which terminations count as a failure, and why does neither of them restart the service after an operator runs `systemctl stop`?

level: middleimportance: must knowfreq 72%

basics

~20 s

Restart=on-failure restarts only after an unclean exit — a nonzero exit status, a fatal signal, a timeout or a watchdog trip. Restart=always additionally restarts after a clean exit 0. Neither applies to an operator-issued systemctl stop, which is an explicit job, not a failure.

open as a page

`systemctl start myapp.service` returns immediately and `systemctl status` reports active (running), but the application does not accept connections for another twenty seconds. What does "active" actually mean under the default Type=simple, and how do you make systemd wait for real readiness?

level: middleimportance: should knowfreq 52%

basics

~20 s

Under Type=simple, systemd calls a unit active as soon as it has forked the main process — it never asks whether the program is ready, or even whether the binary exists. Use Type=exec to wait for a successful exec, or Type=notify so the service reports READY=1 itself.

open as a page

A systemd service configured with Restart=always is sitting in the failed state, and the journal shows "Start request repeated too quickly" followed by a start-limit-hit result. What has systemd decided, which settings control it, and how do you get the unit running again?

level: seniorimportance: should knowfreq 58%

basics

~20 s

systemd's start rate limiter has tripped: the unit was started more than StartLimitBurst= times (5 by default) within StartLimitIntervalSec= (10s by default), so systemd stopped honouring the restart policy and marked the unit failed. Clear the counter with systemctl reset-failed, then fix the crash.

open as a page

On a systemd host, `systemctl stop myapp.service` takes about ninety seconds every time and the journal then reports that systemd killed the process. Walk through what systemd does during a stop job, and which unit settings you would change.

level: seniorimportance: should knowfreq 50%

basics

~20 s

The service is not reacting to SIGTERM, so systemd waits out TimeoutStopSec= — 90 seconds by default — and then sends SIGKILL. A stop job runs ExecStop= if present, signals the unit's processes with KillSignal=, waits, then escalates. Fix the signal handling rather than shortening the timeout.

open as a page

On a systemd host you ran `systemctl stop` and `systemctl disable` on a unit, yet after a reboot it is running again. What does `systemctl mask` do differently, and what does masking look like on disk?

level: middleimportance: nice to knowfreq 42%

basics

~20 s

Disabling only removes the unit's [Install] symlinks, so anything that pulls the unit in as a dependency, or a matching socket, path or timer unit, still activates it. Masking symlinks the unit name to /dev/null under /etc/systemd/system, which makes every activation path fail.

open as a page