skip to content

On an Ubuntu server, `df -h` lists a dozen /dev/loopN filesystems mounted under /snap, every one of them reporting 100% used. What are those mounts, is the 100% a problem, and how do you get a readable df and find where snap disk usage really is?

level: middleimportance: nice to knowfreq 34%

answer

  1. read-only images, not real filesystems
  2. packed exactly full by construction
  3. one loop device per installed revision
  4. the real bytes live under /var/lib/snapd
  5. df -x squashfs for a readable table

basics

~20 s

Those are the read-only squashfs images of installed snap revisions, each attached to a loop device and mounted under /snap. A read-only squashfs is packed exactly full, so 100% is normal and permanent. Filter them with df -h -x squashfs.

solid answer

~40 s

Every installed snap revision ships as a compressed squashfs image in `/var/lib/snapd/snaps/`, and snapd attaches each one to a loop device and mounts it read-only at `/snap/<name>/<revision>` through a generated systemd `.mount` unit. A read-only squashfs contains exactly its own contents with no free space, so `df` reports 100% for every one of them forever — it is not a warning and no monitoring rule should alert on it. To read `df` again, exclude the type: `df -h -x squashfs`. The bytes that actually consume the filesystem are the `.snap` image files under `/var/lib/snapd/snaps/`, which hold several revisions per snap, plus writable data under `/var/snap/`. Retention is controlled with `snap set system refresh.retain=N`.

code

bash · 10 lines
bash
# A df you can actually read
df -h -x squashfs

# Where snap space really goes
sudo du -sh /var/lib/snapd/snaps /var/snap
snap list --all

# Retention: how many revisions are kept per snap
snap get system refresh.retain
sudo snap set system refresh.retain=2

go deeper

for a junior

Recognise that /dev/loop mounts under /snap are installed snaps, not real disks, and that 100% is expected. Know df -h -x squashfs to get a readable table.

for a middle

Explain why a read-only squashfs is always exactly full, that snapd mounts each revision through a generated systemd mount unit, and that actual consumption is the image files under /var/lib/snapd/snaps plus data under /var/snap.

for a senior

Demonstrate the discrimination an on-call engineer needs: filter the noise at the monitoring layer, then check whether /var is genuinely filling from retained revisions, and know the retention setting and its rollback cost before turning it down.

for a principal

Set the standard — how disk alerting is defined fleet-wide so permanently-full read-only mounts never generate pages, and what retention and sizing policy /var gets on hosts where snaps are permitted at all.

## What the loop mounts are A snap is distributed as a single squashfs image — a compressed, read-only filesystem in one file. On install, snapd writes that file to `/var/lib/snapd/snaps/<name>_<revision>.snap`, attaches it to a free loop device, and mounts it at `/snap/<name>/<revision>`. A per-snap `current` symlink points at the revision in use. Because snapd keeps more than one revision of each snap, and because base snaps (`core22`, `core24`), the `snapd` snap itself and the desktop runtimes are all snaps too, even a modest server accumulates a double-digit number of these mounts. The mounts are not ad-hoc: snapd generates systemd mount units for them, named after the mount point with dashes for slashes. You can list them: ```bash systemctl list-units --type=mount | grep snap findmnt -t squashfs ``` ## Why every one reads 100% A squashfs image is built to hold exactly the files it contains and is mounted read-only. There is no free space by construction and there never will be, so `df` computes used/size as 1.00 and prints `100%`. This is the single most reported non-problem in snap operations. The correct response is to stop reading it as a capacity signal — and, on the monitoring side, to exclude squashfs (and usually loop devices generally) from disk-full alerting, because otherwise every Ubuntu host permanently reports full filesystems and the alert becomes noise that trains people to ignore real ones. ## Getting a readable df ```bash df -h -x squashfs ``` `-x <type>` excludes a filesystem type; `-t <type>` is its inverse and shows only that type, which is occasionally useful for auditing. Neither changes anything on disk. `findmnt` and `lsblk` give the same picture in a friendlier shape, and `lsblk` will show the loop devices with their backing files under `/var/lib/snapd/snaps/`. ## Where the space actually goes The loop mounts consume no space of their own; they are views onto files that already exist. Real consumption is in three places: - `/var/lib/snapd/snaps/` — the `.snap` image files, one per retained revision per snap. This is the big one. - `/var/snap/<name>/` — per-revision and common writable system data for each snap. - `~/snap/<name>/` — per-user data, which lives in home directories and therefore lands wherever home is. So the honest measurement is: ```bash sudo du -sh /var/lib/snapd/snaps /var/snap snap list --all ``` `snap list --all` shows every revision snapd is holding, including ones marked `disabled` in the Notes column — those are the retained older revisions kept so a rollback is possible. ## Retention By default snapd retains three revisions of each snap on a classic system (two on Ubuntu Core), pruning the oldest as part of a refresh. The setting is a system-level snap option: ```bash snap get system refresh.retain sudo snap set system refresh.retain=2 ``` The accepted range is 2 to 20. Note the trade: lowering it reclaims disk but shortens how far back you can revert, and two is the floor precisely because keeping one previous revision is what makes rollback possible at all. Setting it does not delete anything immediately — pruning happens on the next refresh of each snap. To reclaim now, remove a specific disabled revision with `snap remove --revision=<N> <name>`, and remove a snap you no longer want entirely with `snap remove --purge <name>` (the `--purge` also discards its data snapshot). ## When it is genuinely a problem Two real failure modes hide behind the noise. First, `/var` filling up: on hosts where `/var` is a small separate filesystem, retained snap revisions plus base snaps can be gigabytes, and that shows in `df` on `/var`, not on the loop mounts. Second, loop device exhaustion is possible in principle on constrained systems, and a snap whose mount unit failed to start leaves `/snap/<name>/current` dangling so the command simply does not run — `systemctl status snap-<name>-<rev>.mount` and `journalctl -u` on that unit tell you which. The interviewing point is discrimination: a page that fires because a read-only image is exactly full is a monitoring defect, while a page for `/var` at 95% on a snap-heavy host is a real one, and they look similar in a `df` you have not filtered.

  • Your monitoring pages every Ubuntu host for a filesystem at 100%. What do you change, and why not just raise the threshold?
    Exclude squashfs and loop-backed mounts from the disk-full check rather than moving the threshold. A read-only squashfs is always exactly 100% full, so no threshold below 100 is meaningful for it, and raising the global threshold would blind you to genuinely full real filesystems. The fix is a filter on filesystem type, applied once in the check definition.
  • Lowering refresh.retain to 2 frees disk. What are you giving up?
    Rollback depth. Retention is what keeps previous revisions on disk, and `snap revert` can only go back to a revision that is still there. At two you keep exactly one previous revision per snap, which covers the usual case of a bad refresh but leaves nothing if two bad refreshes land before you notice. It is also not immediate: pruning happens at the next refresh.
  • A snap command suddenly reports that the file is not found, though snap list still shows it installed. Where do you look?
    At the mount unit for that revision. The executable lives inside a squashfs mounted at /snap/<name>/<revision>, so if the generated systemd mount unit failed, the path behind /snap/<name>/current is empty and the command cannot be found. `systemctl list-units --type=mount | grep snap` and the journal for the failing `snap-<name>-<rev>.mount` unit will show the cause.

saying these in an interview costs you the question

  • Treats the 100% loop mounts as a full disk
  • Tries to free space by unmounting /snap entries
  • Thinks the loop mounts themselves consume disk
  • Measures snap usage with du on /snap
  • Assumes lowering refresh.retain reclaims space immediately

context