skip to content

On a systemd host, `journalctl -b -1` reports that the specified boot does not exist, and the journal only ever covers the current boot. Why is that happening, and how do you make journald keep logs across reboots?

level: middleimportance: must knowfreq 62%

answer

  1. logs die with the boot
  2. tmpfs versus disk
  3. two directories, one setting
  4. Storage=auto depends on a directory
  5. create it, then restart journald

basics

~10 s

The journal is volatile. With the default Storage=auto it lives in tmpfs under /run/log/journal and is discarded at shutdown unless /var/log/journal exists. Create that directory, or set Storage=persistent, then restart systemd-journald.

solid answer

~40 s

journald has two possible homes: `/run/log/journal`, which is tmpfs and therefore dies with the boot, and `/var/log/journal`, which is on disk. The `Storage=` setting in `/etc/systemd/journald.conf` chooses between them — `volatile`, `persistent`, `auto` and `none`. `auto` is the upstream default and means "persistent if `/var/log/journal` already exists, volatile otherwise", so on a distribution that does not ship that directory you silently get a journal that resets every boot. The fix is either `mkdir -p /var/log/journal` (properly: `systemd-tmpfiles --create --prefix /var/log/journal`, which sets the ownership and ACLs) or `Storage=persistent`, which creates the directory itself, followed by `systemctl restart systemd-journald`. After that `--list-boots` and `-b -1` work. Persistence costs disk, so pair it with `SystemMaxUse=` rather than leaving the default cap.

go deeper

for a junior

Know the two locations — /run/log/journal is RAM and /var/log/journal is disk — and that only the second one survives a reboot. Recognise an empty journalctl -b -1 as a storage question, not a broken command.

for a middle

Explain all four Storage= values and why auto hinges on whether the directory already exists, then give the enable recipe: create the directory or set persistent, restart journald, verify with --list-boots.

for a senior

Frame it as evidence preservation: the previous boot is where the OOM kill and the panic live, so a silently volatile journal deletes the data at the moment it matters. Pair persistence with explicit size caps so you do not trade a blind spot for a full /var.

for a principal

Decide it as fleet policy — which node classes keep local history at all, what that costs in disk and in data-at-rest exposure, and how the choice is expressed in the image so it cannot drift back to an accidental default.

## Two homes, one setting `systemd-journald` writes its journal files to exactly one of two locations: - `/run/log/journal/<machine-id>/` — `/run` is a tmpfs, so this is RAM. It is fast, it never touches the disk, and every byte of it disappears when the machine shuts down or crashes. - `/var/log/journal/<machine-id>/` — ordinary disk. Files survive reboots, and `journalctl --list-boots` can enumerate the boots recorded there. Which one is used is decided by `Storage=` in `/etc/systemd/journald.conf`: - `volatile` — always `/run`, never disk. - `persistent` — always disk; journald creates `/var/log/journal` if it is missing, and falls back to `/run` only while `/var` is not yet mounted during early boot. - `auto` — the upstream default and the source of the surprise: disk **if `/var/log/journal` already exists**, otherwise `/run`. - `none` — everything is discarded; only forwarding (for example to syslog) still delivers anything. So a host showing only the current boot is almost always running `Storage=auto` on an image whose `/var/log/journal` directory was never created. Nothing is broken and nothing warns you; the journal has simply been in RAM the whole time. ## Confirming it in ten seconds ```bash ls -d /var/log/journal 2>/dev/null || echo "no persistent journal dir" journalctl --disk-usage # names the path it is measuring journalctl --list-boots # one line means one boot on record systemd-analyze cat-config systemd/journald.conf # effective config + drop-ins ``` That last command matters because `journald.conf` is rarely the whole story: drop-ins under `/etc/systemd/journald.conf.d/*.conf` override the main file, and reading only `/etc/systemd/journald.conf` can leave you arguing about a value that something else has already replaced. ## Turning persistence on Two routes, both without a reboot: ```bash # route A: create the directory with the right ownership and ACLs sudo systemd-tmpfiles --create --prefix /var/log/journal sudo systemctl restart systemd-journald # route B: state the intent in config (survives an image rebuild better) printf '[Journal]\nStorage=persistent\nSystemMaxUse=1G\n' | \ sudo tee /etc/systemd/journald.conf.d/persistent.conf sudo systemctl restart systemd-journald ``` `journalctl --flush` asks journald to move whatever is currently in the runtime journal into the persistent one and continue there; this is the same operation `systemd-journal-flush.service` performs at boot once `/var` is mounted. Because of that flush step, entries produced before `/var` was available are not lost when storage is persistent — they are written to `/run` first and migrated. ## Why anyone cares Everything you want after an unplanned reboot lives in the previous boot: the OOM kill, the kernel panic trail, the last lines a service logged before the box went down. A volatile journal is precisely the configuration that deletes that evidence at the moment it becomes valuable, and the failure is silent — the next-morning investigation just finds an empty `-b -1`. On a host that reboots as part of normal operation, persistence is not a nicety. The counter-argument is real too. Persistence costs disk and write I/O, and on a small root filesystem an unbounded journal is a way to fill `/var`. It also means log data now sits on the node, which matters if entries can contain sensitive payloads. On genuinely ephemeral, immutable nodes whose logs are shipped elsewhere the moment they are written, a volatile journal is a defensible choice — but it should be a decision recorded in config, not an accident of a missing directory. ## The knobs that come with it Once the journal is on disk, cap it: `SystemMaxUse=` (total size of the persistent journal), `SystemKeepFree=` (free space to leave on that filesystem), `SystemMaxFileSize=` (per file), `SystemMaxFiles=`. The `Runtime*` counterparts govern the tmpfs journal. Leaving these unset does not mean unlimited — the defaults are 10% of the filesystem capped at 4 GB, and 15% of the free space respectively — but those defaults are sized for the filesystem, not for what you want to keep. ## A distribution note Do not memorise "Debian is volatile, Fedora is persistent". Images and cloud vendors change this, and the honest answer in an interview is that you check the directory and the effective config on the host in front of you rather than assuming what the distribution defaults to.

  • What does `journalctl --flush` actually do?
    It asks journald to move everything currently held in the runtime journal under `/run/log/journal` into the persistent journal under `/var/log/journal` and carry on writing there. The same operation runs at boot as `systemd-journal-flush.service`, once `/var` is mounted — which is why early-boot entries are not lost on a persistent host even though `/var` was not available when they were produced.
  • Does enabling persistent storage require a reboot?
    No. Create the directory or write the config, then `systemctl restart systemd-journald`; `journalctl --flush` migrates what is already in the runtime journal. Nothing else on the box needs restarting, since journald is the single writer and clients reach it through the socket that stays in place.
  • How do you see the configuration journald is really using?
    `systemd-analyze cat-config systemd/journald.conf` prints the main file together with every drop-in under `/etc/systemd/journald.conf.d/` and the vendor directory, in the order they apply. Reading only `/etc/systemd/journald.conf` is how people end up debugging a setting that a drop-in has already overridden.
  • Is a volatile journal ever the right choice?
    Yes, on genuinely ephemeral nodes whose entries are forwarded off the box as they are produced, or where you deliberately do not want log payloads resting on local disk. The important part is that it be a stated choice with `Storage=volatile`, not the accidental result of a missing `/var/log/journal` on a host you expect to investigate after a crash.

saying these in an interview costs you the question

  • Assumes the journal is always persistent by default
  • Thinks a reboot is needed to enable persistence
  • Says journald writes /var/log/syslog
  • Believes Storage=auto means persistent everywhere
  • Enables persistence without capping journal size

context