skip to content

journald Logging

Reading the journal properly: filtering by unit, priority and time window, following live output, persistent versus volatile storage, rotation and size limits, and the structured fields behind every entry. Interviewers ask because show me the logs for this failing service since the last boot should be a single journalctl invocation.

on this pageshow

questions

5

On a systemd-managed Linux host, a service unit has just failed. Which journalctl filters do you combine to show only that unit's messages, only errors, and only the current boot — and what does each flag do?

level: juniorimportance: must knowfreq 82%

answer

  1. one command, four filters
  2. -u also catches PID 1's lines
  3. -b scopes to this boot
  4. -p is a threshold, not equality
  5. -f follows, --since windows

basics

~20 s

Combine journalctl -u nginx.service to select the unit, -b to scope to the current boot, and -p err to keep priority error and above. Add -f to follow live, -n 100 for the last lines, or --since for a time window.

solid answer

~40 s

One invocation answers it: `journalctl -u nginx.service -b -p err`. `-u` selects a unit and matches both the messages the service itself logged and the lines PID 1 logged about it (started, failed, entered failed state), and it can be repeated to OR several units. `-b` limits the output to the current boot; `-b -1` is the previous one. `-p` is a **threshold**, not an equality test — `-p err` shows priorities 0 through 3, so `crit`, `alert` and `emerg` are included, and `-p 4..6` gives an explicit range. For a time window I add `--since "1 hour ago"` or `--since "2026-08-20 09:00"` with an optional `--until`. `-f` follows live, `-n 200` prints the tail, `-e` jumps to the end, and `-o short-iso` gives unambiguous timestamps.

go deeper

for a junior

Be able to type journalctl -u <unit> -b without pausing, and know that -f follows new entries. Say plainly that the journal is a binary store queried by journalctl, not a text file you tail.

for a middle

Explain that -p is a severity threshold rather than an exact match, that -u also surfaces the service manager's own lines about the unit, and how --since/--until bound a window.

for a senior

Show the diagnosis, not the flags: scope to the boot, widen the priority when a filter comes back empty, and recognise that stdout text lands at info priority so -p err can hide the very error you are hunting.

for a principal

Own the consequence for the fleet: whether services log with meaningful priorities at all decides if a severity filter is usable during an incident, and that is a service-contract decision, not a per-host one.

## What you are querying `systemd-journald` is the log collector on a systemd host. It receives kernel messages, anything a service writes to stdout and stderr (units get `StandardOutput=journal` by default), classic syslog API calls, and native structured log submissions. It stores them in indexed binary files rather than plain text, which is exactly why you query them with `journalctl` instead of `grep`ing a file: the fields are indexed, so filtering by unit or priority is a lookup, not a scan. ## Selecting the unit: -u `journalctl -u nginx.service` restricts output to one unit. The `.service` suffix is optional when it is unambiguous. Two things arrive under that filter: the process's own output, and the lines the service manager itself emitted about the unit — `Starting…`, `Started…`, `Main process exited, code=exited, status=1`, `Failed with result 'exit-code'`. That second group is usually where the verdict lives, which is why `-u` beats matching the raw `_SYSTEMD_UNIT=` field when you are diagnosing a failure. `-u` is repeatable: `journalctl -u nginx -u php-fpm` interleaves both, in time order, which is the cheapest way to correlate two cooperating services. ## Scoping to a boot: -b `-b` means "this boot". `-b -1` is the previous boot, `-b -2` the one before that, and `--list-boots` enumerates what is available. This matters after a crash or a reboot loop: without it you are reading a mixture of boots and can mistake yesterday's stack trace for today's. Past boots exist only if the journal is stored persistently; on a host with a volatile journal, `-b -1` reports that the boot does not exist. ## Priority: -p is a threshold The journal carries the syslog priority scale: 0 emerg, 1 alert, 2 crit, 3 err, 4 warning, 5 notice, 6 info, 7 debug. `-p err` (or `-p 3`) means "err and anything more severe", i.e. 0–3. The single most common mistake is reading it as "exactly err" and then wondering where the `crit` lines went — they were shown, higher up. A range is written `-p 4..6`. There is a real trap here. A program that just prints `ERROR: cannot connect` to stdout is recorded at priority `info`, because that is the default level systemd applies to unprefixed stdout/stderr lines (`SyslogLevel=` in the unit governs it). So `-p err` can show nothing at all while the log is full of the word ERROR. Either drop the priority filter, or make the service log natively with a real priority — a line may also carry a kernel-style `<3>` prefix on stdout, which systemd strips and turns into the priority. ## Time and volume `--since` and `--until` accept absolute stamps (`"2026-08-20 09:00:00"`), calendar words (`yesterday`, `today`), and relative English (`"1 hour ago"`, `"30 min ago"`). `-n 200` prints the last 200 entries, `-e` opens the pager at the end, `-f` follows new entries like `tail -f`, and `-f` combines with every other filter — `journalctl -u api -p warning -f` is the standard "watch this service misbehave" command. For scripts, add `--no-pager`; for timestamps you can paste into a ticket, `-o short-iso` or `-o short-iso-precise` prints ISO-8601 with the timezone offset instead of the local short form. ## Putting it together ```bash # post-mortem on a unit that failed during this boot journalctl -u api.service -b -p warning -o short-iso --no-pager # what happened in the ten minutes around the incident journalctl -u api.service --since "2026-08-20 09:05" --until "2026-08-20 09:15" # tail it live while you reproduce journalctl -u api.service -f -n 50 ``` ## Related flags worth knowing `-k` restricts to kernel messages (the `dmesg` equivalent, but with the boot filter available). `-x` appends catalog explanations to well-known systemd messages. `--user` reads the calling user's own journal instead of the system one. `-t` filters by syslog identifier, which is how you find output from something that is not a unit at all. ## Why interviewers ask "Show me the logs for this failing service since the last boot" should be one command, typed without hesitation. A candidate who dumps the whole journal into `grep`, or who cannot scope to a boot, will spend an incident scrolling.

  • Does `-p err` show only entries whose priority is exactly err?
    No — `-p` sets a maximum numeric priority, so `-p err` returns 0 through 3: `emerg`, `alert`, `crit` and `err`. If you truly want one level, use a range with the same endpoints, `-p 3..3`. Read the flag as "this severity and worse", the same way syslog thresholds have always worked.
  • How would you look at the boot before the current one, and when does that fail?
    `journalctl -b -1`, with `--list-boots` to see what is on disk. It fails when the journal is volatile: with no `/var/log/journal` directory the journal lives in tmpfs under `/run/log/journal` and is discarded at shutdown, so only the current boot exists. Persistent storage is a prerequisite for any cross-reboot forensics.
  • A service prints error text to stdout, but `journalctl -u app -p err` shows nothing. Why?
    Unprefixed stdout and stderr lines are recorded at the unit's `SyslogLevel=`, which defaults to `info`, so a priority filter of `err` excludes them regardless of what the text says. Drop the `-p` filter, set `SyslogLevel=` appropriately, or have the application log natively with real priorities — a `<3>` prefix on a stdout line also works.

saying these in an interview costs you the question

  • Thinks -p err hides crit and emerg entries
  • Greps the whole journal instead of filtering by unit
  • Believes journald writes plain text under /var/log
  • Assumes -b -1 works on a volatile journal
  • Says -f cannot be combined with -u or -p

context

open as a page

On a systemd host, `journalctl -b -1` reports that the specified boot does not exist, and the journal only ever covers the current boot. Why is that happening, and how do you make journald keep logs across reboots?

level: middleimportance: must knowfreq 62%

basics

~10 s

The journal is volatile. With the default Storage=auto it lives in tmpfs under /run/log/journal and is discarded at shutdown unless /var/log/journal exists. Create that directory, or set Storage=persistent, then restart systemd-journald.

open as a page

In the systemd journal, what is the difference between a field such as `_SYSTEMD_UNIT=` and one such as `PRIORITY=`, and how do you filter journalctl output by field and display every field of an entry?

level: middleimportance: should knowfreq 44%

basics

~20 s

Every journal entry is a set of key=value fields. Names starting with an underscore, like _PID and _SYSTEMD_UNIT, are added by journald from the sender's verified credentials and cannot be forged; the rest, such as PRIORITY and MESSAGE, come from the client.

open as a page

On a systemd host, /var is nearly full and /var/log/journal accounts for several gigabytes. How do you reclaim the space now with journalctl, and how do you cap the journal so it cannot grow that far again?

level: seniorimportance: should knowfreq 52%

basics

~10 s

Measure with journalctl --disk-usage, then run journalctl --rotate followed by --vacuum-size= or --vacuum-time=, because vacuuming only removes archived files. Cap growth permanently with SystemMaxUse= and SystemKeepFree= in journald.conf and restart systemd-journald.

open as a page

A busy service's log lines go missing from the systemd journal during traffic spikes, and the journal itself contains a line saying messages from that unit were suppressed. What is systemd-journald doing, and which settings control it?

level: seniorimportance: nice to knowfreq 30%

basics

~20 s

journald rate-limits each service separately: past roughly 10000 messages in 30 seconds it drops the remainder and logs a suppression notice. Tune RateLimitIntervalSec= and RateLimitBurst= in journald.conf, or LogRateLimitIntervalSec= and LogRateLimitBurst= on the unit itself.

open as a page