You are handed a shell on an unfamiliar Linux server. Where do you expect a service's configuration, its persistent state and its log files to live, and what is the rule that puts those three things in different top-level directories?

juniormust knowfreq 70%

answer

three directories, three lifetimes
who writes it, and how long it lives
static package code versus local edits
back up two of them, reinstall the third
read-only root depends on this split

basics

~20 s

Configuration lives under /etc, persistent state under /var/lib and logs under /var/log. The split is by mutability and ownership: /usr holds static package-owned code, /etc holds locally editable configuration, /var holds data the machine writes while it runs.

solid answer

~40 s

The Filesystem Hierarchy Standard splits the tree by who writes a file and how long it lives, not by what kind of file it is. /usr is static and owned by the package manager — binaries, libraries and shipped data, nothing the running system modifies, which is why it can be mounted read-only. /etc is host-local configuration: text, editable by the administrator, small, and the thing that makes this machine different from an identical one. /var is everything the machine itself writes: /var/log for logs, /var/lib/<service> for persistent state such as a database's data files, /var/spool for queues. So on an unknown host I look in /etc/<name> first, then /var/lib/<name> and /var/log/<name>. The payoff is practical: back up /etc and /var, reinstall /usr, and the machine comes back.

The Filesystem Hierarchy Standard splits the tree by **who writes a file and how long it lives**, not by what kind of file it is. `/usr` is static and owned by the package manager — binaries, libraries and shipped data, nothing the running system modifies, which is why it can be mounted read-only. `/etc` is host-local configuration: text, editable by the administrator, small, and the thing that makes this machine different from an identical one. `/var` is everything the machine itself writes: `/var/log` for logs, `/var/lib/<service>` for persistent state such as a database's data files, `/var/spool` for queues. So on an unknown host I look in `/etc/<name>` first, then `/var/lib/<name>` and `/var/log/<name>`. The payoff is practical: back up `/etc` and `/var`, reinstall `/usr`, and the machine comes back.

go deeper

for a junior

Be able to name the three places without hesitating: /etc for configuration, /var/log for logs, /var/lib for a service's data. Say plainly that /usr is program files installed by packages, not user home directories.

for a middle

Explain the organising rule rather than reciting directories: static and package-owned versus local and editable versus written by the running system. Be ready to place an unfamiliar file correctly by applying that rule out loud.

for a senior

Show the operational payoff: which directories a backup must cover, why /var on its own filesystem stops a runaway log from wedging a host, and how the split enables a read-only /usr. Distinguish /var/lib from /var/cache when planning recovery.

for a principal

Own the standard across a fleet: where local convention diverges from the FHS, what that costs when images become immutable, and how to make configuration ship as vendor defaults in /usr with overrides in /etc so upgrades never prompt.

The rule, not the list

The Filesystem Hierarchy Standard (FHS 3.0, maintained by the Linux Foundation; also summarised in the hier(7) man page) is often taught as a list of directories to memorise. That is the wrong way in. The tree is organised along two axes, and once you know them you can place a file you have never seen before.

The first axis is shareable vs unshareable: can this data be identical across many machines, or is it specific to this host? The second is static vs variable: does it change only when an administrator installs or upgrades software, or does the running system write it by itself?

	static	variable
shareable	`/usr`	`/var/mail`, `/srv`
unshareable	`/etc`, `/boot`	`/var/log`, `/var/lib`, `/run`

/usr — static, package-owned code

/usr holds the operating system's programs and data: /usr/bin for executables, /usr/lib for shared libraries and private helper files, /usr/share for architecture-independent data (documentation, icons, locale data, man pages), /usr/include for headers. Every file here is owned by a package. Nothing the system does at runtime modifies it. That property is what makes read-only-root images, atomic-update distributions and shared or verified /usr trees possible.

The name is a historical accident — it did once hold user home directories — but it is now read as "Unix system resources". Home directories live in /home, and root's home is /root.

/etc — the machine's identity

/etc is host-specific configuration. The FHS is explicit that it must contain no binaries: it is configuration, and it should be small enough that a human can read it. /etc/passwd, /etc/fstab, /etc/hosts, /etc/ssh/sshd_config are what distinguish this host from a freshly installed one.

Many projects now ship their defaults under /usr/lib/<project> or /usr/share/<project> and let /etc hold only the administrator's overrides, often as drop-in files in a *.d directory. That is a refinement of the same rule: vendor content is static and package-owned, so it belongs in /usr; your edit is local, so it belongs in /etc. It also means an upgrade never has to ask "you modified this config file, what should I do?" for files you never touched.

/var — what the machine writes

/var is variable data whose existence the administrator does not manage file by file:

/var/log — log files.
/var/lib/<service> — persistent application state: a PostgreSQL cluster's data directory, a package manager's database, a service's cached credentials. This is the directory people most often misplace.
/var/cache — regenerable data; deleting it must cost only performance, never correctness.
/var/spool — queues (mail, print, cron jobs).
/var/tmp — temporary files that must survive a reboot.

The key discipline is separating /var/lib (lose it and you lose data) from /var/cache (lose it and nothing breaks). That distinction drives backups, disk-pressure cleanup and container volume design.

The remaining ephemera

/run holds runtime state valid only since boot — PID files, Unix sockets, lock files — and is cleared at every boot. /tmp is scratch space with no guarantee of surviving a reboot. /srv is data served by this system to the outside world (web roots, exported trees), and /opt is self-contained third-party software. /boot holds the kernel and initramfs; /dev holds device nodes managed by the kernel.

Why the split earns its keep

Three everyday consequences follow directly:

Backups. /etc and /var/lib are the machine. /usr is reinstallable from packages, /var/cache and /tmp are disposable. A backup policy falls out of the hierarchy for free.
Read-only and immutable systems. Because /usr is static, an image-based or container-based system can mount it read-only and keep only /etc, /var and /run writable. Container images lean on exactly this: the layers are /usr-shaped, the volumes are /var-shaped.
Partitioning and disk pressure. Putting /var on its own filesystem means a runaway log cannot fill the root filesystem and wedge the whole host.

# the three places to look for an unfamiliar service called "foo"
ls /etc/foo /var/lib/foo /var/log/foo

When you are unsure where something you are writing belongs, ask the two questions in order: does the package manager own it (/usr), does an administrator edit it (/etc), or does the program write it itself (/var)?

## The rule, not the list The Filesystem Hierarchy Standard (FHS 3.0, maintained by the Linux Foundation; also summarised in the `hier(7)` man page) is often taught as a list of directories to memorise. That is the wrong way in. The tree is organised along two axes, and once you know them you can place a file you have never seen before. The first axis is **shareable vs unshareable**: can this data be identical across many machines, or is it specific to this host? The second is **static vs variable**: does it change only when an administrator installs or upgrades software, or does the running system write it by itself? | | static | variable | |---|---|---| | shareable | `/usr` | `/var/mail`, `/srv` | | unshareable | `/etc`, `/boot` | `/var/log`, `/var/lib`, `/run` | ## /usr — static, package-owned code `/usr` holds the operating system's programs and data: `/usr/bin` for executables, `/usr/lib` for shared libraries and private helper files, `/usr/share` for architecture-independent data (documentation, icons, locale data, man pages), `/usr/include` for headers. Every file here is owned by a package. Nothing the system does at runtime modifies it. That property is what makes read-only-root images, atomic-update distributions and shared or verified `/usr` trees possible. The name is a historical accident — it did once hold user home directories — but it is now read as "Unix system resources". Home directories live in `/home`, and root's home is `/root`. ## /etc — the machine's identity `/etc` is host-specific configuration. The FHS is explicit that it must contain no binaries: it is configuration, and it should be small enough that a human can read it. `/etc/passwd`, `/etc/fstab`, `/etc/hosts`, `/etc/ssh/sshd_config` are what distinguish this host from a freshly installed one. Many projects now ship their *defaults* under `/usr/lib/<project>` or `/usr/share/<project>` and let `/etc` hold only the administrator's overrides, often as drop-in files in a `*.d` directory. That is a refinement of the same rule: vendor content is static and package-owned, so it belongs in `/usr`; your edit is local, so it belongs in `/etc`. It also means an upgrade never has to ask "you modified this config file, what should I do?" for files you never touched. ## /var — what the machine writes `/var` is variable data whose *existence* the administrator does not manage file by file: - `/var/log` — log files. - `/var/lib/<service>` — persistent application state: a PostgreSQL cluster's data directory, a package manager's database, a service's cached credentials. This is the directory people most often misplace. - `/var/cache` — regenerable data; deleting it must cost only performance, never correctness. - `/var/spool` — queues (mail, print, cron jobs). - `/var/tmp` — temporary files that must survive a reboot. The key discipline is separating `/var/lib` (lose it and you lose data) from `/var/cache` (lose it and nothing breaks). That distinction drives backups, disk-pressure cleanup and container volume design. ## The remaining ephemera `/run` holds runtime state valid only since boot — PID files, Unix sockets, lock files — and is cleared at every boot. `/tmp` is scratch space with no guarantee of surviving a reboot. `/srv` is data served by this system to the outside world (web roots, exported trees), and `/opt` is self-contained third-party software. `/boot` holds the kernel and initramfs; `/dev` holds device nodes managed by the kernel. ## Why the split earns its keep Three everyday consequences follow directly: 1. **Backups.** `/etc` and `/var/lib` are the machine. `/usr` is reinstallable from packages, `/var/cache` and `/tmp` are disposable. A backup policy falls out of the hierarchy for free. 2. **Read-only and immutable systems.** Because `/usr` is static, an image-based or container-based system can mount it read-only and keep only `/etc`, `/var` and `/run` writable. Container images lean on exactly this: the layers are `/usr`-shaped, the volumes are `/var`-shaped. 3. **Partitioning and disk pressure.** Putting `/var` on its own filesystem means a runaway log cannot fill the root filesystem and wedge the whole host. ```sh # the three places to look for an unfamiliar service called "foo" ls /etc/foo /var/lib/foo /var/log/foo ``` When you are unsure where something you are writing belongs, ask the two questions in order: does the package manager own it (`/usr`), does an administrator edit it (`/etc`), or does the program write it itself (`/var`)?

Why does the FHS say /etc must contain no binaries?
`/etc` is meant to be small, host-specific, human-readable configuration that an administrator can read end to end and copy between machines. Executables are package-owned, architecture-dependent and static, so they belong in `/usr/bin` or `/usr/lib`. Keeping binaries out also means `/etc` can be captured in configuration management or version control without dragging along compiled artefacts.
If a system mounts /usr read-only, what still has to be writable for it to work?
`/etc` for configuration changes, `/var` for logs and service state, `/run` for runtime sockets and PID files, and `/tmp` for scratch. Image-based systems that want `/etc` read-only too usually provide it as an overlay or a tmpfs seeded from `/usr`, so local edits live somewhere writable while the shipped defaults stay in the read-only tree.
What is the difference between /var/lib and /var/cache, and why does it matter operationally?
`/var/lib` is authoritative state — deleting it loses data. `/var/cache` is regenerable: deleting it costs only time, never correctness. The distinction drives backups (back up `lib`, skip `cache`) and disk-pressure response (a cleanup script may safely empty `/var/cache`, never `/var/lib`). Services that blur the two make both jobs unsafe.

saying these in an interview costs you the question

Says /usr is where user home directories live
Treats /etc as the place for logs or runtime state
Thinks /var is only for log files
Puts a service's database files under /etc because they are configuration
Backs up /usr and skips /var/lib

You are handed a shell on an unfamiliar Linux server. Where do you expect a service's configuration, its persistent state and its log files to live, and what is the rule that puts those three things in different top-level directories?

§ answer

basics

solid answer

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The rule, not the list

/usr — static, package-owned code

/etc — the machine's identity

/var — what the machine writes

The remaining ephemera

Why the split earns its keep

▾common follow-ups3 questions

▾red flags5 flags

§ context

▾related questions6 questions

▾related topics4 topics

The rule, not the list

/usr — static, package-owned code

/etc — the machine's identity

/var — what the machine writes

The remaining ephemera

Why the split earns its keep

answer

go deeper

context