On a Linux host, what is the relationship between the `sysctl` command and the /proc/sys directory, and why does a value set with `sysctl -w` disappear after a reboot?

## One namespace, two spellings The kernel's tunable parameters live in a single tree exposed at /proc/sys. The `sysctl` command from procps-ng is a convenience layer over that tree, and the mapping is purely mechanical: replace each dot with a slash and prefix /proc/sys. These three are the same operation: ``` sysctl net.ipv4.ip_forward sysctl -n net.ipv4.ip_forward cat /proc/sys/net/ipv4/ip_forward ``` and these two are the same write: ``` sysctl -w net.ipv4.ip_forward=1 echo 1 > /proc/sys/net/ipv4/ip_forward ``` `sysctl -a` dumps every key with its current value, which is the practical way to discover what exists on the kernel you are actually running — the set of keys depends on kernel version and on which subsystems are compiled in or loaded. Writing requires root, because /proc/sys entries are owned by root and mode 0644 for read-only-to-users keys or 0600/0644 with write reserved to root. ## Why the change evaporates Procfs stores nothing. When you write `1` into ip_forward, the kernel's handler parses the string and assigns an integer to a variable in kernel memory; there is no file, no block, no inode content. A reboot starts a fresh kernel with its compiled-in defaults, and nothing replays your write. That is the whole answer to the interview question: the write was a command, not a record. ## Making it persist Persistence is a userspace convention layered on top. Configuration files contain plain `key = value` lines and are read at boot: ``` # /etc/sysctl.d/90-forwarding.conf net.ipv4.ip_forward = 1 vm.swappiness = 10 ``` On a systemd distribution, `systemd-sysctl.service` runs early in boot and applies files from `/etc/sysctl.d/`, `/run/sysctl.d/` and `/usr/lib/sysctl.d/`, plus `/etc/sysctl.conf` for compatibility. Precedence has two layers that candidates routinely mix up. Files are processed in lexicographic order of their filename across all the directories, and later assignments win — so `99-local.conf` overrides `10-vendor.conf`. Separately, a file present in `/etc/sysctl.d/` masks a file of the same name in `/usr/lib/sysctl.d/`, which is how you override a package's defaults without editing its file. This is why the numeric prefix convention exists at all. `sysctl -p [file]` loads one file (defaulting to /etc/sysctl.conf), and `sysctl --system` walks the whole set of directories in the same order the boot service uses. Applying with `--system` after editing is the habit that prevents the classic mismatch where the file says one thing and the running kernel says another until somebody reboots months later. ## The two traps **A key only exists if its owner is loaded.** Tunables are registered by the subsystem that owns them, and many subsystems are kernel modules. `net.bridge.bridge-nf-call-iptables`, for example, does not exist until the `br_netfilter` module is loaded, so a sysctl.d file setting it at boot fails with "cannot stat /proc/sys/... : No such file or directory" if it is applied before the module. The fix is to arrange for the module to load first — for instance via a file in /etc/modules-load.d/ — rather than to blame sysctl. The same class of failure explains why a key you can set by hand after the system is up fails during boot. **A value is only consulted when the kernel looks at it.** Some tunables are read continuously and take effect instantly; others are sampled once, at the moment an object is created. `net.core.somaxconn` caps the accept queue that a socket is given when it calls `listen()`, so raising it at runtime does nothing for sockets that are already listening — the server has to create them again. Interviewers like this because it separates people who have actually tuned a machine from people who have read a tuning blog post. Sizes computed at boot from available memory, such as certain default buffer values, are similarly resistant to being changed later. A third detail worth knowing: much of the `net.*` tree is per network namespace rather than global, so the value you set on the host is not automatically what a differently-namespaced process sees. ## Reading a tunable you were asked about Good answers name real keys and say what they do rather than reciting a tuning cargo cult: `net.ipv4.ip_forward` turns the host into a router for IPv4, `vm.swappiness` biases the reclaim balance between page cache and anonymous pages, `fs.file-max` caps system-wide open file descriptors, `kernel.pid_max` sets where PID allocation wraps, and `net.ipv4.ip_local_port_range` bounds the ephemeral ports a client can source from. If you are not sure a key exists on the kernel in front of you, `sysctl -a | grep` answers it in a second — and saying that is a better answer than inventing a plausible-sounding key.