A Linux server refuses to create new files with "No space left on device", yet df reports the filesystem is only 40% full. What is the most likely cause, how do you confirm it, and how do you fix it?
answer
- free space is not the only budget
- count objects, not bytes
- df has a second mode
- ext4 decides at format time
- one inode per file, size irrelevant
basics
~20 sThe filesystem has most likely run out of inodes. Blocks and inodes are separate budgets: an ext4 filesystem fixes its inode count at format time, so a hoard of tiny files can exhaust it while gigabytes of block space remain free. Confirm with df in inode mode.
solid answer
~50 sA filesystem has two independent budgets — data blocks and inodes — and every file, directory, symlink and device node consumes one inode regardless of size. ext2/3/4 allocate the whole inode table when the filesystem is created, based on a bytes-per-inode ratio, so a workload that creates millions of tiny files (session files, spooled mail, cache shards) can exhaust inodes with the block space barely touched. Creating a file then fails with `ENOSPC`, which the shell prints as "No space left on device". `df -i` shows the inode counts and `IUse%`; a filesystem at 100% there with plenty of free blocks confirms it. The fix is to delete or consolidate the small files, because you cannot grow an ext4 inode table in place — only recreating the filesystem with a denser ratio, or growing the filesystem, adds inodes. XFS, which allocates inodes dynamically, avoids the failure mode.
go deeper
Remember that every file and directory costs one inode no matter how small, and that a filesystem can run out of inodes while still having free space.
Explain that ext4 fixes its inode table at format time from a bytes-per-inode ratio while XFS allocates dynamically, and that both exhaustion modes surface as the same ENOSPC error.
Show the full incident path: distinguish inode exhaustion from reserved blocks and from deleted-but-open files, locate the directory hoarding entries, and state which remedies actually work on a live ext4 filesystem.
Own it as a capacity decision — filesystem choice and format parameters for many-small-file workloads, retention and expiry policy for session and cache trees, and monitoring that alerts on inode usage rather than bytes alone.
## Two budgets, not one Every filesystem tracks two separate resources. **Data blocks** hold file contents. **Inodes** hold file metadata — type, mode, owner, timestamps, link count and where the data lives. Each file, directory, symlink, FIFO and device node costs exactly one inode, whether it is empty or a terabyte. Running out of either one produces the same errno, `ENOSPC`, and therefore the same misleading message. This matters because different filesystems manage the inode budget very differently. **ext2/3/4** create the entire inode table at `mkfs` time and can never extend it in place. `mke2fs` sizes it from a bytes-per-inode ratio (defaults come from `/etc/mke2fs.conf`, typically one inode per 16 KiB of space). So a 500 GB ext4 filesystem might get roughly 32 million inodes — plenty for normal files averaging tens of kilobytes, and nowhere near enough for a directory tree of 100-byte files. **XFS** allocates inodes dynamically as files are created, bounded by `imaxpct`, the maximum percentage of filesystem space that inodes may occupy (25% by default, adjustable with `xfs_growfs -m`). Btrfs and ZFS also allocate metadata dynamically. Practically, only the ext family hits classic inode exhaustion, which is why this question so often comes with an ext4 server attached. **tmpfs** is a third case: it has its own `nr_inodes` mount option, so a `/tmp` or `/run` on tmpfs can exhaust inodes independently of its size limit. ## Confirming it `df -i` reports the same filesystems in inode terms: `Inodes`, `IUsed`, `IFree`, `IUse%`. A mount point sitting at `IUse% 100` while ordinary `df` shows plenty of free space is conclusive. Then find the hoard. Inodes are consumed by *entries*, so count files rather than bytes — walking the tree and tallying entries per directory pinpoints it quickly. It is almost always one of a small set of culprits: PHP or application session files, a mail spool or queue directory, an unrotated per-request cache, orphaned temporary files from a crashed job, or a build/dependency tree unpacked once per deploy and never cleaned. A related trap: because directories consume inodes too, a tree of millions of empty directories is just as fatal as one of millions of empty files. ## Fixing it Short term, the only lever is to reduce the number of entries: - Delete the accumulated small files, ideally through the application's own cleanup path so it does not immediately recreate them. - Consolidate: archive many small files into a few large ones, or move that workload to a store designed for many small objects. - Add and enforce expiry — a scheduled cleanup for session and cache directories, and rotation for anything append-only. Long term, on ext4 you cannot raise the inode count on a fixed-size filesystem; `tune2fs` has no option for it. The real remedies are to recreate the filesystem with a denser ratio (`mkfs.ext4 -i <bytes-per-inode>` or `-N <count>`, or a `-T` usage type tuned for many small files), to **grow** the filesystem — growing ext4 with `resize2fs` does add inodes proportionally — or to move that workload onto XFS. ## The other reasons ENOSPC appears with free space An interviewer often follows up here, and a strong answer names the alternatives: - **Reserved blocks.** ext4 reserves a percentage of the filesystem (5% by default) for the superuser so that a full disk does not stop root from repairing it and does not fragment the tail of the filesystem. `df` counts that reservation as free-ish, but an unprivileged process hits `ENOSPC` before it. `tune2fs -m` adjusts the percentage. - **A deleted file still held open.** Blocks stay allocated while a process holds the descriptor — but there `df` shows *full*, not 40%, so the symptom differs. - **Something writing under a mount point.** If a service wrote into a directory before its filesystem was mounted over it, the visible mount looks empty while the hidden data fills the parent. - **Quotas** produce `EDQUOT`, not `ENOSPC`, which is a genuinely different error worth distinguishing. The reasoning pattern is the point: `ENOSPC` says "a filesystem resource is exhausted", and blocks are only one of the resources it could mean.
- Besides inode exhaustion, what else makes a Linux filesystem report ENOSPC while showing free space?Reserved blocks are the common one: ext4 keeps a percentage (5% by default) for root, so unprivileged writes fail first — `tune2fs -m` tunes it. A deleted-but-still-open file also consumes blocks, though there df shows the filesystem full rather than half empty. A directory written to before its filesystem was mounted over it hides data in the parent. Disk quotas are a different error entirely: EDQUOT, not ENOSPC.
- Why can XFS not usually be pushed into this failure, and what is its equivalent limit?XFS allocates inodes dynamically as files are created rather than reserving a fixed table at format time, so there is no pre-set ceiling to exhaust. Its bound is `imaxpct`, the maximum share of filesystem space inode metadata may occupy — 25% by default and adjustable. In practice you run out of blocks long before that, which is why the classic inode-exhaustion incident is an ext-family story.
- Can you increase the inode count of an existing ext4 filesystem?Not on a filesystem that stays the same size — the inode table is laid out at mkfs time and no tune2fs option extends it. The two real options are to grow the filesystem, since growing ext4 with resize2fs allocates additional inode tables proportionally, or to back up the data and recreate the filesystem with a denser bytes-per-inode ratio using mkfs.ext4 -i or -N.
saying these in an interview costs you the question
- Assumes ENOSPC can only ever mean data blocks are full
- Believes tune2fs can raise the inode count in place
- Thinks large files, not many small ones, exhaust inodes
- Says empty files and directories cost no inode
- Treats df -i output as a duplicate of ordinary df