In the systemd journal, what is the difference between a field such as `_SYSTEMD_UNIT=` and one such as `PRIORITY=`, and how do you filter journalctl output by field and display every field of an entry?
answer
- records, not lines
- the underscore means something
- who filled the field in
- same field ORs, different fields AND
- verbose and json show everything
basics
~20 sEvery journal entry is a set of key=value fields. Names starting with an underscore, like _PID and _SYSTEMD_UNIT, are added by journald from the sender's verified credentials and cannot be forged; the rest, such as PRIORITY and MESSAGE, come from the client.
solid answer
~40 sA journal entry is not a line of text but a record of fields. Fields whose names begin with an underscore — `_PID`, `_UID`, `_COMM`, `_EXE`, `_CMDLINE`, `_SYSTEMD_UNIT`, `_BOOT_ID`, `_TRANSPORT` — are **trusted**: journald derives them from the credentials of the socket peer, so the sending process cannot lie about them. Fields without the underscore, such as `MESSAGE`, `PRIORITY`, `SYSLOG_IDENTIFIER` and `MESSAGE_ID`, are supplied by the client and are only as truthful as the client. You filter with bare `FIELD=value` arguments: `journalctl _SYSTEMD_UNIT=nginx.service _PID=1234`. Repeating the same field ORs the values; different fields AND together; a `+` between groups is an explicit OR. `-o verbose` prints every field of every entry, `-o json-pretty` gives machine-readable output, `-N` lists the field names present and `-F <FIELD>` lists all values a field has taken.
go deeper
Know that each journal entry is a set of key=value fields and that -o verbose reveals them. Being able to run journalctl _COMM=sshd and explain what it matched is enough at this level.
Explain the underscore convention — journald fills those in from the sender's credentials, the client supplies the rest — and state the match algebra: same field ORs, different fields AND, + ORs groups.
Use the field model in an investigation: match on trusted fields when attribution matters, know why -u returns more than the raw unit match, and use -F to inventory what has been logging on an unfamiliar host.
Treat structured fields as an interface: deciding that services attach a correlation-id field, and that operators query it rather than grepping message text, is what makes host-local logs usable without a search tier.
## The journal is a database of records The mental model that makes journald click is that it does not store log *lines*, it stores *records*. Each record is a set of `KEY=value` pairs, one of which happens to be `MESSAGE=`, the human-readable text. Everything `journalctl` does — filtering by unit, by boot, by priority — is a lookup over the indexed fields of those records, which is why the tool is fast on a multi-gigabyte journal where `grep` would not be. ## Trusted versus client-supplied fields The naming convention carries a security meaning that is easy to miss: - **Underscore-prefixed fields are added by journald itself.** When a process submits an entry over the journal socket, journald asks the kernel for the peer's credentials and fills in `_PID`, `_UID`, `_GID`, `_COMM`, `_EXE`, `_CMDLINE`, `_SYSTEMD_UNIT`, `_SYSTEMD_CGROUP`, `_BOOT_ID`, `_MACHINE_ID`, `_HOSTNAME`, `_TRANSPORT` and friends from what the kernel reports, not from what the sender claims. A process cannot label its entries as coming from another unit. - **Fields without the underscore came from the client.** `MESSAGE`, `PRIORITY`, `SYSLOG_IDENTIFIER`, `SYSLOG_FACILITY`, `MESSAGE_ID`, and any custom field an application chooses to attach, are whatever the sender wrote. A misconfigured or hostile process can claim `PRIORITY=7` for a catastrophe, or set `SYSLOG_IDENTIFIER=sshd` while being nothing of the sort. The operational consequence: when you need to be certain which unit or which binary produced an entry, match on `_SYSTEMD_UNIT=` or `_EXE=`, not on the identifier string in the message. `_TRANSPORT=` is the same kind of evidence — it tells you whether the entry arrived from the kernel, from stdout, from the syslog socket, or from the native journal API. ## Filtering by field Any bare `FIELD=value` argument to `journalctl` is a match: ```bash journalctl _SYSTEMD_UNIT=nginx.service journalctl _UID=33 _TRANSPORT=stdout journalctl _COMM=sshd --since "today" ``` The combination rules are worth committing to memory, because they are not what people guess: - **Same field, repeated → OR.** `journalctl _SYSTEMD_UNIT=a.service _SYSTEMD_UNIT=b.service` shows entries from either unit. It does not "keep the last one" and it is not a contradiction. - **Different fields → AND.** `journalctl _SYSTEMD_UNIT=a.service PRIORITY=3` shows error-priority entries from that one unit. - **`+` → explicit OR between whole groups.** `journalctl _SYSTEMD_UNIT=a.service + _UID=1000` shows entries matching either side. This is also the difference between `journalctl -u foo.service` and `journalctl _SYSTEMD_UNIT=foo.service`. The `-u` option expands into several matches — including the messages PID 1 logged *about* the unit, and coredump records associated with it — while the raw field match returns strictly what processes inside the unit sent. When `-u` shows a `Failed with result 'exit-code'` line that the field match does not, that is why. ## Seeing the fields at all The default output format shows only a syslog-style summary line. To see the record: ```bash journalctl -u api.service -n 1 -o verbose # every field, human-readable journalctl -u api.service -n 1 -o json-pretty # every field, machine-readable journalctl -N # field names present in the journal journalctl -F _SYSTEMD_UNIT # every value that field has taken ``` `-F` is quietly one of the most useful: `journalctl -F _SYSTEMD_UNIT` is an instant inventory of everything that has logged on the box, and `journalctl -F _COMM` finds the name of a process you only half remember. `-N` (`--fields`) answers "what custom fields is this application actually attaching?" without guessing. ## Custom fields An application using the native journal API — or `logger` and `systemd-cat` at the shell — can attach arbitrary fields, and they become queryable like any other. A service that adds `REQUEST_ID=` to its entries turns "find everything for this request on this host" into `journalctl REQUEST_ID=abc123`. Two constraints: field names must be uppercase alphanumerics with underscores and may not start with an underscore (that namespace is reserved for the trusted fields), and only a subset of fields is indexed, so a match on an unusual field is a scan rather than a lookup and will be slower on a large journal. ## Why interviewers ask this It separates people who treat `journalctl` as a fancier `tail` from people who understand what they are querying. The field model is also what makes the trust question answerable: asked "how do you know this entry really came from that service?", the good answer names the underscore-prefixed fields and where they come from.
- Why does `journalctl -u foo.service` sometimes show lines that `journalctl _SYSTEMD_UNIT=foo.service` does not?`-u` is not a single field match. It expands into several, so it also picks up the messages the service manager itself logged about the unit — starting, started, main process exited, entered failed state — and associated coredump records. The raw field match returns only entries whose sender was actually a process in the unit, which is precisely the set that excludes PID 1's verdict on the failure.
- Can an application forge the unit its entries appear under?No. `_SYSTEMD_UNIT` and the other underscore fields are filled in by journald from the socket peer's credentials as reported by the kernel, so a client cannot set them. It can, however, set `SYSLOG_IDENTIFIER` or the message text to anything at all — which is why identification during an incident should rest on the trusted fields, not on the name printed in the line.
- How do you get a list of every service that has logged on a host?`journalctl -F _SYSTEMD_UNIT` prints every distinct value that field has taken across the journal. The companion `-N` lists the field *names* present, which is how you discover the custom fields an application is attaching without reading its source.
saying these in an interview costs you the question
- Thinks the journal stores plain text lines
- Believes an app can set its own _SYSTEMD_UNIT
- Says repeating a field ANDs the two values
- Treats -u as identical to a _SYSTEMD_UNIT match
- Assumes fields are invisible unless the app prints them