skip to content

You put a service behind a systemd socket unit listening on port 8080, and now the service fails at startup with "address already in use" — yet nothing else on the host uses that port. What is going on, and how do you confirm it?

level: seniorimportance: should knowfreq 30%

answer

  1. something does hold the port
  2. check the owner, not just the port
  3. PID 1 in the users field
  4. activation is opt-in for the daemon
  5. a way to test without unit files

basics

~20 s

systemd already holds the listening socket, and the daemon is still calling bind() on the same port instead of using the descriptor it was handed. Something does occupy 8080 — PID 1. Either the daemon must adopt the passed descriptor, or drop the socket unit.

solid answer

~40 s

There *is* something on port 8080: systemd. The socket unit opened the listener before the service ever started, and the daemon ignored the passed descriptor and tried to bind the port for itself, which fails with `EADDRINUSE`. Confirm it with `ss -tlnp 'sport = :8080'` — the listener will show `users:(("systemd",pid=1,...))` — and with `systemctl list-sockets`, which names the socket unit and the unit it triggers. Then establish whether the binary supports activation at all: run it standalone under `systemd-socket-activate -l 8080 /usr/local/bin/myapp` and see whether it serves on the inherited descriptor or tries to bind again. If it has no support, you have two honest options — patch it to read `LISTEN_FDS` and use fd 3, or delete the socket unit and let the daemon bind the port itself as a plain service.

code

bash · 9 lines
bash
# who is actually listening on 8080?
ss -tlnp 'sport = :8080'

# which socket unit is it, and what does it trigger?
systemctl list-sockets --all
systemctl status myapp.socket

# does the binary pick up a passed descriptor at all?
systemd-socket-activate -l 8080 /usr/local/bin/myapp

go deeper

for a junior

Know that under a socket unit the port is held by systemd, so the daemon must not bind it itself. Being able to check the owner of a listening port is the skill being tested.

for a middle

Explain that activation is opt-in: the daemon must read the passed descriptors instead of creating its own, and that failing to do so is exactly what produces the bind error. Name the variables that carry the handover.

for a senior

Work the evidence in order — owner of the listener, which unit owns the socket, whether the binary supports the protocol at all — and then choose between patching the daemon and dropping the socket unit. Recognise the mirror failure when the service is started with the socket stopped.

for a principal

Treat it as an adoption-cost question: socket activation requires source-level support in every service you convert, so before mandating it across a fleet, know which daemons already implement it and what the fallback story is for the ones that never will.

## The contradiction is only apparent "Nothing else uses the port" is usually based on looking for another copy of the application. Under socket activation the process holding the port is `systemd` itself — PID 1 opened it when the socket unit started, and it holds it permanently. Any process that then calls `bind()` on the same address gets `EADDRINUSE`, exactly as it would against any other listener. So the diagnosis is not "a stale process" and not "a lingering socket": it is a daemon that was placed behind a socket unit without implementing the handover. ## Confirming it in thirty seconds ```bash # who holds the listener? ss -tlnp 'sport = :8080' # which unit is it, and what does it activate? systemctl list-sockets --all systemctl status myapp.socket systemctl status myapp.service # look for TriggeredBy: ``` The giveaway is the `users:` field in the `ss` output naming `systemd` at `pid=1`. If instead it names your own daemon, the socket unit is not the culprit and you are looking at an ordinary port conflict — a second copy, a container publishing the port, or a leftover process. That branch is worth stating out loud, because it shows you are actually reading the evidence rather than pattern-matching. ## Establishing whether the binary supports activation The activation contract is opt-in: a program only benefits if it looks for `LISTEN_FDS`, checks `LISTEN_PID`, and uses the descriptors starting at fd 3 instead of creating its own socket. There is no way for systemd to force this on a program that does not implement it. The quickest test needs no unit files: ```bash systemd-socket-activate -l 8080 /usr/local/bin/myapp ``` This opens the socket, sets up the same environment systemd would, and runs the binary. If it serves requests, the program supports activation. If it fails with the same bind error, it does not. Checking the project's documentation for socket activation, or grepping the binary for the variable names, tells you the same thing more crudely. ## The three ways out **Patch the daemon.** The change is small — take the descriptor when one is passed, and fall back to binding when none is. Many daemons already have this and it just needs enabling in their config. **Drop the socket unit.** If the daemon cannot be changed, run it as a plain service that binds the port itself: `systemctl disable --now myapp.socket`, then enable the service. You lose on-demand start and restart continuity, but you get a working service, and a working service beats an elegant broken one. **Change the endpoint.** Occasionally the right answer is that the socket unit and the daemon should listen on different things — for instance systemd holding a Unix socket for local clients while the daemon binds its own TCP port. That is a design change, not a fix, and should be deliberate. ## The mirror-image failure The same misunderstanding produces a second symptom that is worth recognising. Start the service directly with `systemctl start myapp.service` while the socket unit is stopped, and no descriptors are passed at all: `LISTEN_FDS` is unset and the count comes back zero. A daemon that assumes it will always be handed a socket then exits immediately with something like "no sockets passed". The cure is the same fallback path — bind normally when nothing was passed — which is also what makes the binary runnable by hand during development. ## Why this question separates people It is a small puzzle whose answer requires holding two facts at once: that a socket unit opens the port before the service exists, and that the daemon must cooperate to use it. Someone who has only read about socket activation tends to assume it is transparent to the application. Someone who has deployed it knows the first thing you check is whether the daemon supports the protocol at all.

  • The ss output names your own daemon rather than systemd. What does that change about the diagnosis?
    It rules socket activation out entirely — you have a plain port conflict. Look for a second copy of the service, a container publishing the same port onto the host, or a process left behind by a previous start. The socket unit may still be misconfigured, but it is not what is holding 8080, and the fix is on the duplicate-listener side instead.
  • How do you make a daemon usable both standalone and socket-activated?
    Give it the fallback: ask how many descriptors were passed, use them when the count is positive, and create and bind the socket yourself when it is zero. That single branch keeps the binary runnable from a shell during development and under a socket unit in production, and it avoids the second failure mode where starting the service directly aborts because nothing was handed over.
  • Is there any way to socket-activate a daemon that has no support for the protocol?
    Not directly — the daemon has to accept the descriptors, and nothing can force that. The usual workaround is to put something that does support it in front, so the activated process is a small proxy that relays to the real daemon on a different address. That adds a hop and a component to operate, so it is only worth it when on-demand start genuinely matters.

saying these in an interview costs you the question

  • Blames a stale process or a lingering socket state
  • Says systemd should have released the port before starting the service
  • Assumes socket activation is transparent to the application
  • Kills PID 1's listener by restarting systemd
  • Never checks which process actually owns the listening socket

context