On a systemd host you edited /usr/lib/systemd/system/nginx.service directly, and a package upgrade silently reverted your change. Which directories does systemd load unit files from, in what order of precedence, and how should that change have been made so it survives upgrades?

## Why this comes up Almost every real systemd change is a change to somebody else's unit file: raise a file-descriptor limit on the vendor's nginx unit, add an environment variable to a database unit, tighten a sandbox setting. Where you write that change decides whether it survives the next package upgrade and whether the next engineer can find it. Interviewers ask this because the wrong answer produces a change that quietly vanishes months later. ## The unit search path The system manager (PID 1) reads unit files from a list of directories with a defined priority. From highest to lowest, the ones that matter day to day are: - `/etc/systemd/system` — administrator-owned. Nothing but you writes here. - `/run/systemd/system` — runtime, on tmpfs. Generators and transient units land here; it disappears on reboot. - `/usr/lib/systemd/system` — package-owned. Your distribution's `.rpm`/`.deb` files install units here. On distributions that have merged `/usr`, `/lib/systemd/system` is a compatibility symlink to the same place. There is also `/usr/local/lib/systemd/system` for locally built software, and generator output directories under `/run/systemd/`. You can print the effective list on a host with `systemd-analyze unit-paths`. The rule for the *main* unit file is winner-takes-all: systemd looks for `nginx.service` in each directory in order and uses the first one it finds. A full copy in `/etc` does not merge with the vendor copy — it replaces it entirely. That is occasionally what you want, but it means you have permanently forked the unit and will not receive upstream changes to it. ## Drop-ins: the mechanism you should be using Beside a unit file, systemd looks for a directory named after the unit with a `.d` suffix, and reads every `*.conf` inside it: ``` /etc/systemd/system/nginx.service.d/override.conf /usr/lib/systemd/system/nginx.service.d/50-vendor.conf ``` Drop-ins are *additive*: the main unit file is parsed first, then each drop-in is applied on top, in lexicographic order of the file names across all directories, with `/etc` applied after `/usr/lib` so the administrator wins on any setting both touch. A drop-in that sets only `[Service] LimitNOFILE=65535` leaves everything else in the vendor unit exactly as it was. Because the file names are sorted, the convention of prefixing them (`10-`, `50-`, `99-`) lets configuration-management tools slot changes in predictably. Drop-ins also exist for whole unit types — a `.conf` under `/etc/systemd/system/service.d/` applies to every service unit on the host — which is handy for fleet-wide defaults and surprising when you have forgotten it exists. ## The commands - `systemctl edit nginx.service` opens an editor on `/etc/systemd/system/nginx.service.d/override.conf`, creating it if needed, and runs a daemon-reload when you save. - `systemctl edit --full nginx.service` instead copies the whole vendor unit into `/etc/systemd/system/` for editing — the deliberate fork. - `systemctl cat nginx.service` prints the main unit file and every drop-in, each preceded by a comment naming its absolute path. This is the fastest way to answer "what is this unit actually configured to do". - `systemctl show nginx.service` prints the fully merged properties as systemd resolved them, which is the ground truth when a drop-in and a main file disagree. - `systemctl revert nginx.service` removes administrator drop-ins and copies, restoring the vendor unit. - `systemd-delta` walks the whole host and reports which units are overridden, extended by drop-ins, or masked. ## daemon-reload systemd parses unit files once and keeps them in memory. Any hand edit — a new file, a changed drop-in, a deleted override — requires `systemctl daemon-reload` before it has any effect; until then `systemctl status` may warn that the unit file changed on disk. Reloading the manager does **not** restart anything: the new configuration applies to the running unit only where systemd can apply it live, and a process-level setting like `ExecStart=` or `User=` takes effect at the next restart of that unit. ## What actually went wrong in the scenario Unit files are not treated as configuration files by the packaging tools, so there is no conffile prompt and no `.rpmsave` backup: the upgrade simply wrote a new `/usr/lib/systemd/system/nginx.service` over the old one, and the change was gone with no warning. The drop-in in `/etc` would have been untouched, and `systemctl cat` would still show it applied on top of the new vendor unit.