You are handed a shell on an unfamiliar Linux server. Where do you expect a service's configuration, its persistent state and its log files to live, and what is the rule that puts those three things in different top-level directories?
answer
- three directories, three lifetimes
- who writes it, and how long it lives
- static package code versus local edits
- back up two of them, reinstall the third
- read-only root depends on this split
basics
~20 sConfiguration lives under /etc, persistent state under /var/lib and logs under /var/log. The split is by mutability and ownership: /usr holds static package-owned code, /etc holds locally editable configuration, /var holds data the machine writes while it runs.
solid answer
~40 sThe Filesystem Hierarchy Standard splits the tree by **who writes a file and how long it lives**, not by what kind of file it is. `/usr` is static and owned by the package manager — binaries, libraries and shipped data, nothing the running system modifies, which is why it can be mounted read-only. `/etc` is host-local configuration: text, editable by the administrator, small, and the thing that makes this machine different from an identical one. `/var` is everything the machine itself writes: `/var/log` for logs, `/var/lib/<service>` for persistent state such as a database's data files, `/var/spool` for queues. So on an unknown host I look in `/etc/<name>` first, then `/var/lib/<name>` and `/var/log/<name>`. The payoff is practical: back up `/etc` and `/var`, reinstall `/usr`, and the machine comes back.
go deeper
Be able to name the three places without hesitating: /etc for configuration, /var/log for logs, /var/lib for a service's data. Say plainly that /usr is program files installed by packages, not user home directories.
Explain the organising rule rather than reciting directories: static and package-owned versus local and editable versus written by the running system. Be ready to place an unfamiliar file correctly by applying that rule out loud.
Show the operational payoff: which directories a backup must cover, why /var on its own filesystem stops a runaway log from wedging a host, and how the split enables a read-only /usr. Distinguish /var/lib from /var/cache when planning recovery.
Own the standard across a fleet: where local convention diverges from the FHS, what that costs when images become immutable, and how to make configuration ship as vendor defaults in /usr with overrides in /etc so upgrades never prompt.
## The rule, not the list The Filesystem Hierarchy Standard (FHS 3.0, maintained by the Linux Foundation; also summarised in the `hier(7)` man page) is often taught as a list of directories to memorise. That is the wrong way in. The tree is organised along two axes, and once you know them you can place a file you have never seen before. The first axis is **shareable vs unshareable**: can this data be identical across many machines, or is it specific to this host? The second is **static vs variable**: does it change only when an administrator installs or upgrades software, or does the running system write it by itself? | | static | variable | |---|---|---| | shareable | `/usr` | `/var/mail`, `/srv` | | unshareable | `/etc`, `/boot` | `/var/log`, `/var/lib`, `/run` | ## /usr — static, package-owned code `/usr` holds the operating system's programs and data: `/usr/bin` for executables, `/usr/lib` for shared libraries and private helper files, `/usr/share` for architecture-independent data (documentation, icons, locale data, man pages), `/usr/include` for headers. Every file here is owned by a package. Nothing the system does at runtime modifies it. That property is what makes read-only-root images, atomic-update distributions and shared or verified `/usr` trees possible. The name is a historical accident — it did once hold user home directories — but it is now read as "Unix system resources". Home directories live in `/home`, and root's home is `/root`. ## /etc — the machine's identity `/etc` is host-specific configuration. The FHS is explicit that it must contain no binaries: it is configuration, and it should be small enough that a human can read it. `/etc/passwd`, `/etc/fstab`, `/etc/hosts`, `/etc/ssh/sshd_config` are what distinguish this host from a freshly installed one. Many projects now ship their *defaults* under `/usr/lib/<project>` or `/usr/share/<project>` and let `/etc` hold only the administrator's overrides, often as drop-in files in a `*.d` directory. That is a refinement of the same rule: vendor content is static and package-owned, so it belongs in `/usr`; your edit is local, so it belongs in `/etc`. It also means an upgrade never has to ask "you modified this config file, what should I do?" for files you never touched. ## /var — what the machine writes `/var` is variable data whose *existence* the administrator does not manage file by file: - `/var/log` — log files. - `/var/lib/<service>` — persistent application state: a PostgreSQL cluster's data directory, a package manager's database, a service's cached credentials. This is the directory people most often misplace. - `/var/cache` — regenerable data; deleting it must cost only performance, never correctness. - `/var/spool` — queues (mail, print, cron jobs). - `/var/tmp` — temporary files that must survive a reboot. The key discipline is separating `/var/lib` (lose it and you lose data) from `/var/cache` (lose it and nothing breaks). That distinction drives backups, disk-pressure cleanup and container volume design. ## The remaining ephemera `/run` holds runtime state valid only since boot — PID files, Unix sockets, lock files — and is cleared at every boot. `/tmp` is scratch space with no guarantee of surviving a reboot. `/srv` is data served by this system to the outside world (web roots, exported trees), and `/opt` is self-contained third-party software. `/boot` holds the kernel and initramfs; `/dev` holds device nodes managed by the kernel. ## Why the split earns its keep Three everyday consequences follow directly: 1. **Backups.** `/etc` and `/var/lib` are the machine. `/usr` is reinstallable from packages, `/var/cache` and `/tmp` are disposable. A backup policy falls out of the hierarchy for free. 2. **Read-only and immutable systems.** Because `/usr` is static, an image-based or container-based system can mount it read-only and keep only `/etc`, `/var` and `/run` writable. Container images lean on exactly this: the layers are `/usr`-shaped, the volumes are `/var`-shaped. 3. **Partitioning and disk pressure.** Putting `/var` on its own filesystem means a runaway log cannot fill the root filesystem and wedge the whole host. ```sh # the three places to look for an unfamiliar service called "foo" ls /etc/foo /var/lib/foo /var/log/foo ``` When you are unsure where something you are writing belongs, ask the two questions in order: does the package manager own it (`/usr`), does an administrator edit it (`/etc`), or does the program write it itself (`/var`)?
- Why does the FHS say /etc must contain no binaries?`/etc` is meant to be small, host-specific, human-readable configuration that an administrator can read end to end and copy between machines. Executables are package-owned, architecture-dependent and static, so they belong in `/usr/bin` or `/usr/lib`. Keeping binaries out also means `/etc` can be captured in configuration management or version control without dragging along compiled artefacts.
- If a system mounts /usr read-only, what still has to be writable for it to work?`/etc` for configuration changes, `/var` for logs and service state, `/run` for runtime sockets and PID files, and `/tmp` for scratch. Image-based systems that want `/etc` read-only too usually provide it as an overlay or a tmpfs seeded from `/usr`, so local edits live somewhere writable while the shipped defaults stay in the read-only tree.
- What is the difference between /var/lib and /var/cache, and why does it matter operationally?`/var/lib` is authoritative state — deleting it loses data. `/var/cache` is regenerable: deleting it costs only time, never correctness. The distinction drives backups (back up `lib`, skip `cache`) and disk-pressure response (a cleanup script may safely empty `/var/cache`, never `/var/lib`). Services that blur the two make both jobs unsafe.
saying these in an interview costs you the question
- Says /usr is where user home directories live
- Treats /etc as the place for logs or runtime state
- Thinks /var is only for log files
- Puts a service's database files under /etc because they are configuration
- Backs up /usr and skips /var/lib