Linux internals & storage

A classic LVM snapshot owns a fixed copy-on-write area and preserves old chunks by copying them into it. A thin snapshot just shares block references in the thin pool: a write allocates a new pool block, nothing is copied, and the cost does not grow with the number of snapshots.

The sysctl is kernel.perf_event_paranoid, exposed at /proc/sys/kernel/perf_event_paranoid. Higher values restrict unprivileged use: 2 permits user-space measurement only, 1 also allows kernel profiling, -1 removes restrictions. Granting CAP_PERFMON is the privileged alternative.

strace -c aggregates instead of printing every call: one row per system call with call count, error count and time. It answers which syscalls dominate and which fail, at the cost of losing the sequence in which they happened.

Incrementing a value fetched from a shared map is a non-atomic read-modify-write, so CPUs running the hook concurrently overwrite each other's counts. __sync_fetch_and_add() makes the update atomic; a per-CPU map gives each CPU its own value, and user space sums across CPUs.

XDP_TX resends the packet out the interface it arrived on; XDP_REDIRECT hands it to a different target through a redirect map. Encapsulation with direct server return lets backends reply straight to the client, so only the request half crosses the load balancer.

XDP attaches in native mode inside the driver, generic mode in the shared kernel receive path after the sk_buff is allocated, or offloaded onto a capable NIC. Generic mode is a correctness fallback with none of the performance benefit — the driver did not support native XDP.

Attachments made through a bpf_link end when the last file descriptor to that link closes, while netlink XDP attachments, tc filters and cgroup attachments hold their own kernel reference and outlive the loader. Those have to be detached explicitly.

fentry and fexit programs attach through a BPF trampoline at the compiler-inserted patch site and read typed arguments derived from BTF. An fexit program sees the function's arguments and its return value together; a kretprobe sees only the return.

That kernel was built without CONFIG_DEBUG_INFO_BTF, so it exposes no BTF and libbpf has nothing to resolve the binary's CO-RE relocations against. Options are a kernel built with BTF enabled, supplying an externally generated BTF for that exact kernel, or falling back to a runtime-compiled bcc tool.

Inside a container the loading fails because the bpf and perf_event_open system calls are usually blocked by the default seccomp profile and the required capabilities are dropped. From the host, the PIDs printed are the host's, and the file paths belong to the container's mount namespace, so neither matches what the container shows.

The verifier counts instructions it simulates across every reachable path, not instructions in the program, so branches and loops multiply the work. Reduce path explosion: cut branches, help its bounds tracking, move iteration into bpf_loop(), and split logic into independently verified global subprograms or tail calls.

Both work. A whole-disk PV is simpler and self-aligning but leaves no partition table, so other tools and humans can mistake the disk for blank. A partitioned PV advertises its use through the Linux LVM type code and lets the disk be shared, at the cost of an extra layer to manage.

Confirm the remaining physical volumes hold enough free extents, then run pvmove /dev/sdb to relocate its extents onto them while the volumes stay online. Finish with vgreduce to drop the disk from the group and pvremove to clear its LVM label.

lvconvert --merge schedules the snapshot's preserved contents to be written back over the origin, reverting it to the snapshot's point in time and deleting the snapshot when finished. If the origin is open — mounted or in use — the merge is deferred until the volume is next deactivated and reactivated.

Once the pool has no free blocks, any write needing a new allocation fails or hangs, filesystems on the thin volumes go read-only or corrupt, and metadata exhaustion is worse — the pool goes read-only and needs offline repair. Prevent it with Data%/Meta% alerting, autoextend, and volume group headroom.

Plain -g means frame-pointer unwinding, and optimised x86-64 builds usually compile with -fomit-frame-pointer, freeing that register for general use — so there is no chain to walk. --call-graph dwarf copies part of the user stack per sample and unwinds it offline using DWARF debug data; --call-graph lbr uses the CPU's branch records.

Root inside a container runs with a reduced capability set that normally omits CAP_SYS_PTRACE, so the ptrace attach is refused. Either grant that capability when the container starts, or run strace from the host against the process's host PID.

DPDK takes the NIC away from the kernel for maximum throughput, at the cost of the whole kernel network stack, its tooling, and dedicated busy-polling cores. XDP stays inside the kernel, slightly slower but selective, interrupt-driven and operable with normal tools. Choose by how much of the traffic is exceptional.

Standardise on libbpf CO-RE binaries for anything shipped, because they need no compiler or headers on production hosts, and keep bcc for ad-hoc work on lab machines. Then fix a minimum kernel and BTF baseline, a pinned build toolchain, and a load-test matrix covering every kernel in the fleet.

tcpdump reads packets from an AF_PACKET socket that is fed later in the receive path, after the driver has built an sk_buff. Native XDP runs before that, so packets it drops never reach the tap. Count drops in a BPF map instead.

It generates a C header that embeds the compiled BPF object and declares a struct named after it, with open, load, attach and destroy functions plus typed members for every map, program, link and global variable — replacing hand-written libbpf boilerplate and file loading at runtime.

/dev/dm-2 is the real device-mapper device node, whose number is assigned at activation and is not stable. /dev/mapper/vg0-data and /dev/vg0/data are udev-created symlinks to it. Dashes inside a VG or LV name are doubled in the /dev/mapper form.

Low instructions-per-cycle with heavy cache misses means the CPU is stalling on memory rather than doing work — the fix is data layout and access patterns. A high IPC means the core is genuinely retiring instructions, so the code is doing too much work, and you optimise the algorithm.

ltrace traces calls into shared libraries — malloc, strcmp, OpenSSL functions — which never appear in a system-call trace. It works by breakpointing dynamic-linking stubs, so a statically linked or Go binary that makes no such calls gives an empty trace.

A BPF hash map has a fixed max_entries; once full, inserting a new key fails with -E2BIG and, if the program ignores the return value, the event is silently dropped. BPF_MAP_TYPE_LRU_HASH instead evicts an approximately least-recently-used entry so inserts keep succeeding.

A cgroup program attaches to a cgroup v2 directory and governs every process inside it; a BPF LSM program attaches to a kernel security hook. In both cases the return value decides whether the kernel allows the operation to proceed.

A uprobe patches a breakpoint instruction into a private copy of the target's executable page, so every thread that reaches that address traps into the kernel and runs the BPF program. Each hit costs roughly a microsecond, far more than a tracepoint. Stripped, statically linked, Go and JIT-compiled binaries make it unreliable.

The snapshot is crash-consistent: it looks exactly like the volume would after a power cut, so a journalling filesystem mounts it after replaying its journal, but anything still buffered above the block layer is missing. fsfreeze -f flushes and quiesces the filesystem first, giving a cleanly-shut-down image instead.

The verifier proves architectural safety, not speculative-execution safety, and it is itself a large piece of attack surface whose bugs have been privilege escalations. Distributions therefore disable unprivileged loading and expect eBPF tooling to run with CAP_BPF plus the capability its program type needs.

Allocate lean and grow on demand. LVM growth is online and cheap, while shrinking is offline at best and impossible on XFS, so unallocated extents are the reversible position. Free extents also give operations like pvmove somewhere to relocate data.