You run setcap cap_net_bind_service=+ep on /usr/local/bin/api and it binds port 80 correctly. After the release pipeline copies that same binary to another host with tar, the service fails with a permission error on bind. Where did the capability go?

## What setcap actually writes A file capability is not a permission bit. `setcap` serialises a small structure — a permitted set, an inheritable set, and an effective flag — and stores it in the extended attribute `security.capability` on the file's inode. `getcap` is just a pretty-printer for that attribute; `getfattr -n security.capability /usr/local/bin/api` shows the raw bytes. Because it is an extended attribute rather than part of the mode, `ls -l` shows nothing unusual. A binary with `cap_net_bind_service=ep` looks exactly like any other `0755` file, which is the first reason this trips people up. ## Why the attribute disappears Extended attributes in the `security` namespace are not carried along by default. Every one of these produces a file without the capability: - `cp src dst` — copies data and, with `-p`, mode, ownership and timestamps, but not extended attributes. You need `--preserve=xattr` (or `-a`, which implies it). - `tar` without `--xattrs` on both create and extract. Most CI pipelines tar an artifact directory with default flags. - `install`, `rsync` without `-X`, `scp`, a container image build step, `git checkout`, or a package manager unpacking a payload that never recorded the attribute. - Anything that replaces the file rather than modifying it: recompiling, re-linking, or an editor that writes a temp file and renames it over the original. The capability was on the old inode, which is now unlinked. On top of that, setting the attribute is itself privileged: it requires `CAP_SETFCAP`, normally meaning root. So an xattr-preserving copy run as an unprivileged user still cannot recreate it — the copy either fails to set it or silently ends up without it. ``` sudo setcap cap_net_bind_service=+ep ./api cp ./api /tmp/api-copy getcap /tmp/api-copy # prints nothing sudo cp --preserve=xattr ./api /tmp/api-kept getcap /tmp/api-kept # /tmp/api-kept cap_net_bind_service=ep ``` ## The two environment traps Even a correctly copied file can be inert. First, the destination filesystem must be able to store security-namespace attributes. ext4, XFS and btrfs do. A typical NFS mount, a `vfat` USB stick, and some network or overlay setups do not, and the `setcap` call fails outright or the attribute never survives. Second, the mount options matter. `nosuid` tells the kernel not to honour set-user-ID and set-group-ID bits *or file capabilities* for programs executed from that filesystem. If `/usr/local` or a data volume is mounted `nosuid` — common for volumes holding uploaded or user-writable content — the binary executes with no capability and no error message at exec time; you only see the failure when the privileged operation is attempted. `mount | grep nosuid` is the check. ## Reading the failure correctly The symptom in the question is the classic one: it works where you set it, fails where you deployed it, and the binaries are byte-for-byte identical. That is the signature of metadata loss rather than a code or configuration difference. The diagnostic sequence is short — compare `getcap` on both hosts, then `mount` for `nosuid`, then check the filesystem type. It is also worth knowing that file capabilities are keyed to the file itself, so a package upgrade that replaces the binary silently reverts it. If the capability was applied by hand once, the next update breaks the service and nobody remembers why. ## Where the grant belongs The durable answer is that a file capability is deployment state, not build state. Apply it where the file lands: - from the package's post-install script, so upgrades reapply it; - from a configuration-management resource that asserts the capability on the path; - or avoid the file capability entirely and have the service manager grant the capability to the process at start-up, so the binary on disk stays ordinary and nothing about copying it matters. The last option is usually preferable at fleet scale: it keeps privilege out of the artifact, survives rebuilds and reinstalls automatically, and leaves one place — the service definition — where a reviewer can see what the process is allowed to do.