skip to content

What is the Linux kernel ring buffer that the `dmesg` command prints, and why might messages from early boot be missing from it on a host that has been running for weeks?

level: juniorimportance: should knowfreq 46%

answer

  1. a circle, not a growing file
  2. fixed size, oldest entries lost
  3. it lives in RAM only
  4. timestamps count from kernel start
  5. persistence is somebody else's job

basics

~20 s

It is a fixed-size circular buffer in kernel memory holding messages the kernel prints. When it fills, the oldest entries are overwritten, so on a long-running or chatty host the boot-time messages have usually been discarded.

solid answer

~50 s

Kernel code logs through `printk`, which writes into a circular buffer of fixed size in kernel memory; `dmesg` just reads that buffer out. Because the size is fixed — set when the kernel is built and overridable with `log_buf_len=` on the kernel command line — the buffer wraps: new messages overwrite the oldest ones. On a host with noisy hardware or a firewall logging packets, boot messages can be gone within hours. The buffer is also per-boot, living only in RAM, so a reboot loses it entirely. If you need the boot messages later you have to read the copy that the logging daemon persisted to disk rather than the live buffer. Two other gotchas: the timestamps are seconds since the kernel started, not wall-clock time, and on many distributions unprivileged users cannot read the buffer at all because `kernel.dmesg_restrict` is enabled.

go deeper

for a junior

Be able to say that dmesg prints a fixed-size in-memory kernel buffer that wraps, so the oldest messages — including boot ones — are eventually overwritten and a reboot clears it entirely.

for a middle

Explain that kernel code logs via printk into that buffer, that its size is set at build time and adjustable on the kernel command line, and why persistence requires a user-space daemon.

for a senior

Show judgment about which source to trust in an incident: the live buffer for what is happening now, persisted kernel logs for correlating a past event, and neither if the console showed nothing at all.

for a principal

Own the policy: whether persistent logging is enabled fleet-wide, whether the buffer is sized for hosts that log heavily, and how kernel messages reach central logging before the buffer wraps.

## What it is The kernel cannot call a logging library — it runs before any user-space logger exists and must be able to log while holding locks, in interrupt context, or while a device is failing. So it writes into a **ring buffer**: a fixed-size block of kernel memory treated as a circle. The function kernel code calls is `printk`, and `dmesg` is the user-space command that reads the buffer's contents out and prints them. "Fixed size" is the whole story behind most surprises here. The size is chosen when the kernel is compiled and can be raised at boot with the `log_buf_len=` kernel command-line parameter. When the buffer is full, the next message overwrites the oldest one — there is no growth and no eviction policy to tune beyond making it bigger. ## Why boot messages disappear On a machine that has just started, the buffer holds exactly what you want: firmware handover details, CPU and memory setup, driver probes, the root filesystem mount. But every later kernel message competes for the same space. A host with a flapping link, a disk reporting errors, or a packet filter logging drops can generate thousands of lines an hour, and the buffer wraps. Weeks later, `dmesg` shows the last few minutes of kernel chatter and nothing about the boot at all. The buffer is also **per-boot**. It lives in RAM and is not written anywhere by the kernel itself, so a reboot loses everything in it. Persistence comes from a user-space logging daemon that reads the buffer and writes to disk. That means the live buffer answers "what has the kernel said recently", while the on-disk logs answer "what did the kernel say during the last boot" — and those are genuinely different questions. ## The timestamp trap The numbers in brackets at the start of each line are **seconds since the kernel started**, not a date. The kernel does not know the wall-clock time when it prints its first messages. `dmesg -T` converts them for you by adding the boot time, which is convenient and slightly unreliable: the conversion assumes a steady clock, so entries can be reported at the wrong wall-clock time on a host that suspended or had its clock stepped by time synchronisation. When correlating a kernel message with an application event, prefer the persisted logs, which carry real timestamps. ## Who can read it Many distributions ship with `kernel.dmesg_restrict` enabled, which limits reading the buffer to processes with the appropriate privilege. The symptom is a plain permission error from `dmesg` as an ordinary user, on a system where everything else works — not a broken command, a deliberate hardening setting, because kernel messages leak addresses and hardware details useful to an attacker. ## Filtering what you get Every `printk` carries a severity level, so you can ask for only the serious lines rather than reading thousands: ```bash dmesg --level=err,crit,alert,emerg # only the messages that indicate trouble dmesg -T # approximate wall-clock timestamps ``` ## Why this matters at boot time The ring buffer is the only narration you get for the stage between the kernel taking control and user space starting. If a machine is failing to boot, the messages on the console *are* this buffer being echoed as it is written. And the converse is the diagnostic rule worth remembering: if a machine shows no kernel messages at all on the console, the failure happened before the kernel was running — in firmware or the bootloader — and the ring buffer will never have anything to say about it.

  • How would you get at the kernel messages from a boot that happened last week?
    Not from the live buffer — it holds only the current boot, and probably only its recent minutes. You read the copy a logging daemon persisted to disk, which requires that persistent logging was actually enabled beforehand; many distributions default to keeping logs only for the current boot. On a machine where that matters, enabling persistence is a change you make before the incident, not after.
  • Why are the default dmesg timestamps not wall-clock times?
    Because the kernel starts printing long before it knows the real time — the clock is read from hardware and corrected by user-space time synchronisation much later. Counting seconds from kernel start is something the kernel can always do accurately. `dmesg -T` converts them by adding the estimated boot time, which drifts if the host suspended or the clock was stepped.
  • An ordinary user runs dmesg and gets a permission error, but the system is otherwise healthy. What is going on?
    The `kernel.dmesg_restrict` setting is enabled, which is the default on several distributions. It restricts reading the kernel ring buffer to sufficiently privileged processes because kernel messages expose memory addresses, hardware details and driver internals that help an attacker. The fix is to read it with elevated privilege, not to disable the hardening.

saying these in an interview costs you the question

  • Thinks dmesg reads a log file on disk
  • Expects boot messages to persist across reboots
  • Reads the bracketed numbers as wall-clock time
  • Assumes the buffer grows to fit all messages
  • Treats a permission error as a broken command

context