skip to content

A configuration-management run adds the drop-in /etc/systemd/system/app.service.d/override.conf containing an [Service] section with a new ExecStart= line, and after daemon-reload the unit refuses to start, complaining it has more than one ExecStart=. Why does that happen, and how do you correctly replace a directive from a drop-in?

level: seniorimportance: should knowfreq 38%

answer

  1. drop-ins merge, they do not replace
  2. two kinds of setting, two behaviours
  3. some keys accumulate instead of overwriting
  4. an empty assignment clears the list
  5. order the reset before the value

basics

~20 s

Drop-ins merge rather than replace, and list-valued settings such as ExecStart= append, so the unit ended up with two. Reset the list first by assigning the key an empty value, then set the new one — two lines: ExecStart= followed by ExecStart=/new/command.

solid answer

~40 s

A drop-in is merged over the main unit file, and how a setting merges depends on whether it is single- or list-valued. Single-valued keys such as `User=` or `Type=` are simply overwritten by the last assignment. List-valued keys — `ExecStart=`, `ExecStartPre=`, `Environment=`, `EnvironmentFile=` and others — *append*, so the drop-in's line joined the vendor's rather than replacing it, and a service that is not `Type=oneshot` is refused with more than one `ExecStart=`. The fix is the empty assignment: a bare `ExecStart=` with no value resets the accumulated list, and the next line sets the value you want. The two lines must be in that order, in the same drop-in. Then confirm with `systemctl cat app.service` for the merged file set and `systemctl show -p ExecStart app.service` for the value systemd actually resolved.

code

ini · 3 lines
ini
[Service]
ExecStart=
ExecStart=/usr/local/bin/app --new-flag

go deeper

for a junior

Know that a drop-in adds to the vendor unit rather than replacing it wholesale, and that ExecStart= needs a special reset line before you can change it.

for a middle

Explain the split between single-valued settings, which the last assignment overwrites, and list-valued ones, which accumulate — and the empty-assignment idiom that clears a list.

for a senior

Diagnose it from the journal: read the more-than-one-ExecStart= failure, confirm the merge with systemctl cat, and prove the fix with systemctl show -p rather than trusting the file on disk.

for a principal

Treat it as a class of change failure: require systemd-analyze verify in the pipeline that renders unit drop-ins, and set the rule for when a fork of the vendor unit is preferable to a drop-in built mostly of reset lines.

## Two merge rules, not one When systemd assembles a unit it parses the main unit file, then each drop-in in order. Every assignment it meets is applied to the unit under construction, and the *type of the setting* decides what "applied" means: - **Single-valued settings** hold one value: `User=`, `Type=`, `WorkingDirectory=`, `MemoryMax=`, `RestartSec=`. The last assignment parsed wins, so a drop-in simply overrides the main file. This is the behaviour people generalise from, and it is why the trap is so easy to fall into. - **List-valued settings** accumulate: `ExecStart=`, `ExecStartPre=`, `ExecStartPost=`, `Environment=`, `EnvironmentFile=`, `ReadWritePaths=`, `After=`, `Wants=`, `AmbientCapabilities=`. Each assignment adds to what is already there. So a drop-in containing one `ExecStart=` line does not replace the vendor's `ExecStart=` — it adds a second one. ## Why a second ExecStart= is fatal Several `ExecStart=` lines are legal for a `Type=oneshot` service, where they mean "run these commands in sequence". For every other service type, systemd supervises a single main process and rejects the unit outright rather than guessing which command you meant. The unit fails to load, so it does not just misbehave — it will not start at all, and on the next reboot the host comes up without it. The same shape appears with `Environment=`: two assignments of the same variable are not an error, so the unit starts, and which value wins is decided by parse order rather than by intent. That failure is quieter and lands in production more often. ## The empty assignment systemd's documented reset for a list-valued setting is an assignment with no value at all. A bare key and `=` clears everything accumulated so far for that key; assignments after it start a fresh list. So the correct drop-in is two lines: ```ini [Service] ExecStart= ExecStart=/usr/local/bin/app --new-flag ``` Order matters — the reset must come before the new value, or it wipes what you just set. The reset also only affects the setting it names; nothing else in the merged unit is disturbed. The same idiom clears the other lists: a bare `Environment=` drops all inherited environment assignments, a bare `ReadWritePaths=` clears an inherited sandbox exception list, and so on. It is worth internalising as the single answer to "how do I *remove* something a vendor unit set", because there is no delete syntax beyond it. ## Verifying the merge Two commands settle any argument about what a unit is actually configured to do: ``` systemctl cat app.service systemctl show -p ExecStart -p Environment app.service ``` `systemctl cat` prints the main unit file and every drop-in with its path, in the order they were applied — so you can see the reset line sitting before the new value. `systemctl show -p` prints the property as systemd resolved it after merging, which is the ground truth. If `show` reports two `ExecStart` entries, the reset is missing or misordered, whatever the files look like. Before deploying, `systemd-analyze verify /etc/systemd/system/app.service` loads the unit the way the manager would — including its drop-ins — and reports the problem without disturbing the running service. In a configuration-management pipeline that check belongs in CI, because the failure mode here is a unit that will not start, discovered at the worst possible moment. ## The operational lesson This is a class of incident, not a one-off: the change looks correct in review, the drop-in file is exactly what the ticket asked for, and the unit is dead until someone reads the journal. Three habits prevent it. Know which of the settings you touch are list-valued. Always pair a list-valued override with its reset line. And after any automated push that touches unit files, assert the resolved property with `systemctl show -p` rather than assuming that a written file equals an applied setting. When a change is large enough that half the vendor's `[Service]` section is being reset, the honest move is `systemctl edit --full` to take ownership of the whole unit, and to accept the maintenance cost that comes with the fork — a drop-in full of reset lines is harder to reason about than an explicit copy.

  • Which other unit settings behave the same way as ExecStart= in a drop-in?
    The list-valued ones: `ExecStartPre=`, `ExecStartPost=`, `ExecStopPost=`, `Environment=`, `EnvironmentFile=`, `ReadWritePaths=` and the other sandbox path lists, `AmbientCapabilities=`, and the relationship keys in `[Unit]`. All accumulate across the main file and every drop-in, and all are cleared by an assignment with an empty value.
  • Why does an appended Environment= line not fail the unit the way a second ExecStart= does?
    Because multiple environment assignments are legal — they build one environment block. Setting the same variable twice is not an error, so the unit starts and the last assignment parsed wins. That makes it the more dangerous case: no failure, just a value nobody intended. `systemctl show -p Environment` is the check.
  • When should you abandon drop-ins and take a full copy of the unit instead?
    When the override is mostly resets — if you are clearing and restating a large part of the vendor's `[Service]` section, the drop-in is harder to read than the unit it modifies. `systemctl edit --full` copies the file into /etc and you own it outright, at the cost of tracking upstream changes yourself at upgrade time.

saying these in an interview costs you the question

  • A drop-in always replaces the setting it names
  • ExecStart= behaves the same as User= when overridden
  • Any service may have several ExecStart= lines
  • The reset line can go after the new value
  • Writing the file is proof the setting was applied

context