skip to content

A maintenance script that will run as a systemd service currently opens /var/log/maintenance.log itself, writes its own timestamps and truncates the file when it grows too large. What should a script run under systemd do with its output instead, and how does it mark one line as an error?

level: seniorimportance: should knowfreq 38%

answer

  1. let the supervisor own the logging
  2. stdout is already the log
  3. the manager adds time, unit and PID
  4. stderr is not automatically an error
  5. angle-bracket priority prefix per line

basics

~20 s

A script run under systemd should just print to stdout and stderr; systemd captures both into the journal and adds the timestamp, unit and PID. Both streams get the same default priority, so mark errors with a <3> prefix on the line.

solid answer

~50 s

Under systemd a service's stdout and stderr are captured into the journal by default, and each line becomes an entry stamped with the time, the unit, the PID and the invoking user. So the script's log file, hand-written timestamps and home-grown truncation are all duplicated work — print to stdout and let the manager do it. The one thing that is not automatic is severity: both streams are recorded at the same default priority, so writing to stderr does not make an entry an error. To set it, prefix the line with a syslog priority in angle brackets, such as `<3>` for error or `<4>` for warning; systemd strips the prefix and records the priority. Two practical details follow from stdout being a pipe rather than a terminal: write one event per line, since each line is a separate entry, and remember that programs the script invokes may block-buffer their output when it is not a terminal.

code

bash · 13 lines
bash
#!/usr/bin/env bash
log_info()  { printf '<6>%s\n' "$*"; }
log_warn()  { printf '<4>%s\n' "$*"; }
log_error() { printf '<3>%s\n' "$*" >&2; }

log_info "starting maintenance run"

if ! stdbuf -oL /opt/app/compact-db; then
  log_error "compaction failed, aborting run"
  exit 1
fi

log_info "maintenance run complete"

go deeper

for a junior

Know that a script run as a systemd service should just print to stdout and stderr, and that the timestamps and unit names in the journal are added for you.

for a middle

Explain that both streams share one default priority and that a line-leading angle-bracket level such as <3> is how a script marks an error, or that systemd-cat -p does it for a piped stream.

for a senior

Show the operational reasoning: one event per line because each line is an entry, no self-rotation of a file the process holds open, and awareness that a child's output block-buffers once stdout is a pipe.

for a principal

Own the boundary — the script emits structured, severity-tagged events on stdout and the deployment decides where they go — so the same artefact runs under a service manager, a container runtime and a terminal without conditional logging code.

## The contract A process supervised by systemd is not expected to manage logging. Its standard output and standard error are connected to the journal by default, and every line written there becomes a journal entry. systemd attaches the metadata: the wall-clock and monotonic timestamps, the unit name, the PID, the UID, the boot ID, and more. A script that opens its own file and prefixes each line with a date is re-implementing all of that badly, and it is doing so in a place where the operator's tools cannot see it. The self-rotation is worse than redundant. Truncating a file the process still holds open is a classic way to lose data or leave a sparse file, and the journal already bounds its own disk usage without any help. So the script's logging layer collapses to `echo` and `printf`. ## Severity: the part that is not automatic The common assumption is that stdout means informational and stderr means error. systemd does not work that way — both streams are recorded at the same default priority, informational unless the unit says otherwise. A stack trace on stderr and a progress line on stdout land in the journal looking equally routine, which is why filtering by priority turns up nothing useful for scripts that never set one. The mechanism to set it is a log-level prefix: begin a line with a syslog priority in angle brackets and systemd strips it off and records that priority on the entry. The numbers are the syslog levels — `<0>` emergency through `<7>` debug — with the useful ones being `<3>` error, `<4>` warning and `<6>` info. This is enabled by default and controlled by the unit's `SyslogLevelPrefix=` setting, so a script can rely on it but should tolerate running outside systemd, where the prefix is just visible text. A small helper keeps the call sites clean: ```bash log_info() { printf '<6>%s\n' "$*"; } log_warn() { printf '<4>%s\n' "$*"; } log_error() { printf '<3>%s\n' "$*" >&2; } ``` The alternative, for one-off lines or for a script that also runs by hand, is to pipe through `systemd-cat`, which takes `-t` for a tag and `-p` for a priority and forwards what it reads into the journal. ## Line discipline Each line is one entry. That has two consequences worth designing around. Multi-line output fragments: a ten-line error report becomes ten entries, only the first of which carries your `<3>` prefix, and they can be interleaved with other output. Prefer one self-contained line per event, carrying the context that a reader needs, over a pretty multi-line block. If the payload is genuinely large — a diff, a failing command's full output — write it to a file and log the path. And because the destination is a pipe rather than a terminal, decoration aimed at humans is noise. Guard colour output on the stream actually being a terminal: ```bash if [[ -t 1 ]]; then red=$'\e[31m'; reset=$'\e[0m'; else red=; reset=; fi ``` Without that guard, escape sequences end up embedded in journal entries. ## Buffering The script's own `echo` writes are unbuffered, but the programs it calls are usually not. Many programs choose line buffering when stdout is a terminal and much larger block buffering when it is a pipe — which is exactly the situation under systemd. The visible effect is a long-running child whose output appears in the journal in bursts, or not at all if the process is killed before it flushes. When you need timely output from such a child, run it under `stdbuf -oL` to request line buffering, and be aware that this only works for programs that use the standard C library's default buffering rather than managing their own. ## When a file is genuinely required Sometimes an auditor or an existing shipper needs a plain file. That is a deployment decision expressed in the unit, not something the script should do to itself: the script keeps writing to stdout, and the unit directs that stream where the operator wants it. Keeping the script on the plain-stdout contract is what lets the same script run unchanged from a terminal, from a container, and from a service.

  • Why is the script's own log rotation a problem rather than just redundant?
    Truncating or renaming a file the process still has open is where data goes missing: writes continue at the old offset, leaving a sparse file, or land in a file nobody is reading any more. Bounding log size is the logging system's job, and the journal already enforces its own limits without cooperation from the script.
  • A long-running command the script calls produces nothing in the journal until it exits. Why?
    Its output is a pipe, not a terminal, so it has almost certainly switched from line buffering to block buffering and is holding several kilobytes before flushing. Run it under `stdbuf -oL` to ask for line buffering. If the program manages its own buffering internally, that has no effect and you need the program's own flush option.
  • How do you keep the same script useful when someone runs it by hand in a terminal?
    Keep everything on stdout and stderr and make the priority prefixes the only systemd-specific detail; outside the journal they are just a few visible characters. Guard colour on `[[ -t 1 ]]` so escape sequences never reach a pipe, and avoid anything that assumes a terminal, such as progress bars that rewrite the line.

saying these in an interview costs you the question

  • Thinks writing to stderr makes a journal entry an error
  • Has the script open and rotate its own log file
  • Formats multi-line log blocks that fragment into separate entries
  • Emits colour escape sequences with no terminal check
  • Assumes a child program's output flushes as promptly as the script's

context