skip to content

Linux internals & storage

5 roadmaps60 questionsupdated

The kernel-side subsystems you reach for when ordinary userspace tools stop explaining things: flexible storage with LVM, and in-kernel observability through eBPF and the tracing toolchain. Interviewers go here for senior systems and SRE roles, where the expectation is that you can see inside the kernel rather than only around it.

on this pageshow

guide

overview

~1 min

This hub covers the parts of Linux you reach for once `top`, `df` and the application's own logs stop explaining what is happening: the storage layer that sits between a filesystem and its disks, and the tools that let you watch the kernel at work on a live host. Interviewers bring it into senior systems, SRE and platform rounds because the answers separate people who operate Linux from the outside from people who can reason about what the kernel is doing on their behalf. The hub has three sections. [LVM](/topics/os-lvm) is Linux's volume manager: how disks are pooled, carved into volumes, grown and moved while services keep running, and snapshotted before something risky. [eBPF](/topics/os-ebpf) is the mechanism for running small, kernel-checked programs at hooks across the system, and it underpins much of modern tracing, packet processing and runtime security. [Tracing and profiling tools](/topics/os-tracing-diagnostics) covers the classic instruments, `strace`, `ltrace` and `perf`, that answer "what is this process doing right now?" Junior questions check the layer model: which command touches which layer, and what a tool can see. Senior and principal questions turn into incidents and trade-offs, such as a full snapshot, a tracer that slows production, or a filter that loads but never runs where you expected. Start with LVM's three layers and the classic tracers, then move to eBPF, which takes the tracing ideas inside the kernel.

primer

### Storage is a stack of layers A filesystem sits on a block device, and under LVM that block device is a logical volume built by device-mapper from extents on one or more disks. Each layer keeps its own idea of size. Most LVM questions come down to naming which layer you changed and which one still has to hear about it. ### Growing is cheap; shrinking is not Adding capacity and moving data between disks happen online in LVM. Taking capacity away needs the filesystem's cooperation and on some filesystems is impossible. That asymmetry shapes provisioning policy, and interviewers reward candidates who leave room to grow. ### A snapshot shares the origin's fate LVM snapshots are cheap because they depend on the origin volume for everything that has not changed. That makes them a stable source to copy from and a quick rollback point, not a backup. Classic and thin snapshots pay for this differently, and each fails in its own way when its space runs out. ### Every observer costs something Each tracing tool sits at a different depth and charges a different price: a ptrace-based tracer stops its target on every event it traces, a sampling profiler interrupts on a timer, and an eBPF program runs inline and can summarise in the kernel. Pick the instrument by what it sees and what it costs on a busy host. ### eBPF is verified code in the kernel An eBPF program is checked by the **verifier** before it may run, and its program type decides where it can attach. Maps hold its state and carry results to user space. The verifier explains most of the odd constraints candidates meet: a small stack, bounded loops, null checks, and a fixed menu of helpers. ### Stability is a property of the hook Some attachment points are a declared interface; others are internal functions that the next kernel may rename or inline. Tooling splits the same way, between compiling on every host and shipping one binary that adapts to the kernel it lands on. Fleet questions reward choosing the stable side on purpose.

Physical volume (PV)
A disk or partition initialised for LVM, with a label and metadata area, whose space is divided into extents that a volume group can allocate.
Volume group (VG)
A pool formed from one or more physical volumes. Logical volumes are allocated from its free extents, and it is the unit you extend by adding disks.
Logical volume (LV)
A block device carved from a volume group's extents. Filesystems and databases sit on it, unaware of which physical disks hold its data.
Thin pool
An LVM volume that allocates blocks on first write to the thin volumes it backs, so their combined virtual size can exceed the real space.
Copy-on-write snapshot
A point-in-time view of a volume that stores only the blocks changed since it was taken and reads unchanged blocks from the origin.
ptrace
The kernel interface that lets one process stop, inspect and resume another. Debuggers and strace are built on it, which is why they slow the target.
eBPF verifier
The in-kernel checker that analyses every eBPF program at load time and refuses any it cannot prove safe and terminating.
eBPF map
A kernel-resident data structure that eBPF programs read and write, and that user space reaches through file descriptors. It holds state and carries results out.
kprobe and tracepoint
Two kernel hook styles: a kprobe instruments almost any kernel function dynamically, while a tracepoint is a fixed, named instrumentation point placed in the source.
XDP
eXpress Data Path, an eBPF hook at the earliest point of packet receive, normally inside the driver, that can drop, pass or redirect packets before the normal stack sees them.
BTF and CO-RE
BPF Type Format describes kernel types; Compile Once, Run Everywhere uses it so one compiled eBPF binary adapts to different kernels at load time.

The three sections meet on one host, around one question: where is the time, or the space, going? A request from an application crosses the system-call boundary, passes through a filesystem, lands on a logical volume, and device-mapper turns it into I/O on real disks. Each tool in this hub watches a different stretch of that path. - **LVM** owns the lower half. It decides which disks hold a volume's extents, so capacity, striping, snapshots and disk evacuation are questions about that mapping. - **strace and ltrace** sit at the top, at the system-call and library-call boundaries of a single process. They show what a program asked for and what came back. - **perf** samples what the CPU is executing, user code and kernel alike, and reads the hardware's performance counters. - **eBPF** can attach at most of these points and beyond, from a syscall entry to a block-layer event or a network driver, and aggregate in the kernel instead of copying every event out. An answer that names the layer first and the tool second is usually the one interviewers want. The same disk-heavy service looks different at each depth: ```bash # which volume, and which disks under it lsblk -o NAME,TYPE,SIZE,MOUNTPOINT # what the process asks the kernel for (ptrace: costly on a busy host) strace -f -p "$PID" -e trace=%file # what actually reaches the block layer, counted in the kernel bpftrace -e 'tracepoint:block:block_rq_issue { @[comm] = count(); }' ``` The first line reads the layout, the second watches one process from outside at real cost, and the third summarises across the machine with little overhead.

  1. Physical Volumes, Volume Groups & Logical Volumes →

    The three LVM layers and extents; every resize, snapshot and migration question assumes this model.

  2. Resizing & Migration →

    Growing volumes, adding disks and evacuating a failing one online: the routine operations interviewers walk through step by step.

  3. strace and ltrace →

    The system-call boundary and the simplest live tracer, plus what attaching it costs a production process.

  4. perf and CPU profiling →

    Sampling, call stacks, flame graphs and hardware counters, for when the question becomes where CPU time goes.

  5. Verifier and Safety Model →

    Why eBPF programs are shaped the way they are; most later eBPF constraints trace back to the verifier.

  6. Tracing and Observability →

    Putting eBPF to work on observability, where most interview scenarios for it land.

  • Growing a logical volume and assuming the filesystem grew with it; the two layers are resized separately unless you ask for both.

  • Shrinking in the wrong order, or planning to shrink XFS at all; see Resizing & Migration before touching lvreduce.

  • Calling an LVM snapshot a backup: it lives on the same disks as its origin and is only useful while the origin survives.

  • Creating a snapshot or thin pool without watching how full it gets; running out of space is when both fail, and they fail badly.

  • Attaching strace to a hot production process without mentioning its overhead, or without naming a lower-cost alternative.

  • Reading a flame graph's horizontal axis as time; width means share of samples, and left-to-right order carries no timeline.

  • Building a fleet tool on kprobes of internal kernel functions and not expecting it to break after a kernel upgrade.

  • Assuming an XDP program is fast because it loaded; if it fell back to generic mode, it runs later in the path with little gain.

This guide assumes a current long-term kernel in the 6.x range and LVM2. Most LVM behaviour discussed here has been stable for years; eBPF is where the kernel version changes the answer, and interviewers on mixed fleets ask about it. - **5.3** let the verifier accept loops it can bound, ending the era of mandatory unrolling. - **5.5** added the BPF trampoline behind fentry and fexit programs. - **5.7** added BPF LSM programs. - **5.8** introduced the BPF ring buffer and split eBPF and perf privileges into `CAP_BPF` and `CAP_PERFMON`, so tooling no longer needs full `CAP_SYS_ADMIN`. - **5.17** added the `bpf_loop()` helper. Kernels that lack BTF information cannot run CO-RE binaries without outside help, so "which kernel, built how" is a fair first question before any eBPF answer. When a feature is newer than the oldest kernel in the fleet, say so and name the fallback.

**Storage.** LVM is a volume manager layered under a separate filesystem. ZFS and Btrfs fold pooling, snapshots and checksums into the filesystem itself, at the cost of a different operational model. Software RAID with mdadm and hardware RAID provide redundancy and are often stacked under LVM rather than replacing it. **Observability and packets.** eBPF is the base of bpftrace and bcc for ad hoc tracing, and is used by products such as Cilium for networking and Falco for runtime security. For new tooling it has largely replaced kernel-module approaches such as SystemTap, because the verifier makes in-kernel code safe to load on production hosts. DTrace, from Solaris, is the ancestor most dynamic-tracing ideas are compared to. For packet processing, DPDK bypasses the kernel for raw throughput, while XDP keeps the kernel and its tooling in play. Choosing between them is a principal-level question.

explore

report an issue with this guide →

questions

page 1 of 2

In eBPF, what is a map, and how does a user-space program get at the data that a kernel-side eBPF program writes into one?

level: juniorimportance: must knowfreq 78%

answer

  1. shared key/value store living in the kernel
  2. helpers on one side, syscall on the other
  3. the user-space handle is a file descriptor
  4. copies in and out, not shared memory
  5. refcounted — last reference frees it

basics

~20 s

An eBPF map is a kernel-resident key/value store shared by BPF programs and user space. BPF code uses helpers like bpf_map_lookup_elem(); user space copies values in and out via the bpf() syscall on the map's file descriptor.

solid answer

~50 s

A map is the only way an eBPF program can keep state or hand anything back, because the program itself has no memory that outlives one invocation beyond its small stack. It is a kernel object created with the `bpf()` syscall (`BPF_MAP_CREATE`), fixed at creation time in type, `key_size`, `value_size` and `max_entries`, and identified in user space by a file descriptor. Inside the kernel the BPF program touches it only through helpers — `bpf_map_lookup_elem()`, `bpf_map_update_elem()`, `bpf_map_delete_elem()`. User space issues the matching `bpf()` commands on that descriptor, usually through libbpf wrappers, which **copy** the key in and the value out; it is not a pointer into the program's memory. Because the handle is a file descriptor, the map is reference-counted: when the last descriptor and program reference go away the map is freed, unless it has been pinned into the bpffs filesystem to outlive the loader.

code

c · 26 lines
c
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, __u32);
    __type(value, __u64);
} openat_count SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_openat")
int count_openat(void *ctx)
{
    __u32 pid = bpf_get_current_pid_tgid() >> 32;
    __u64 init = 1;

    __u64 *val = bpf_map_lookup_elem(&openat_count, &pid);
    if (!val) {
        bpf_map_update_elem(&openat_count, &pid, &init, BPF_ANY);
        return 0;
    }
    __sync_fetch_and_add(val, 1);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

go deeper

for a junior

Be able to say plainly that a map is a key/value store in the kernel, that BPF code reaches it with helper calls, and that user space reaches it through a file descriptor.

for a middle

Explain that type, key size, value size and capacity are fixed at BPF_MAP_CREATE, and that user-space lookups copy through the bpf() syscall rather than sharing memory.

for a senior

Show you know the cost model: aggregate in the kernel, poll summaries rather than per-event records, and know that iteration with BPF_MAP_GET_NEXT_KEY is not a consistent snapshot.

for a principal

Own the interface decision — which state lives in maps, which maps are pinned and therefore become a contract between processes, and who is allowed to hold a descriptor to them.

## Why maps exist at all An eBPF program is a short piece of verified bytecode that the kernel runs at a hook point — a kprobe firing, a packet arriving, a tracepoint being hit. It gets a context pointer, a stack of a few hundred bytes, and no ability to write into any process's memory. When it returns, everything local to it is gone. So there has to be somewhere to put results, and somewhere to keep state between invocations. That somewhere is a **map**. A map is a kernel-owned data structure, most commonly a key/value store, that survives across program invocations and is reachable from both the BPF side and user space. Almost every real eBPF tool is structured the same way: a small program in the kernel that filters and aggregates into a map, plus a user-space agent that reads the map and prints, exports, or acts on it. ## The shape is fixed when the map is created Maps are created by the `bpf()` system call with the `BPF_MAP_CREATE` command, which takes: - `map_type` — `BPF_MAP_TYPE_HASH`, `BPF_MAP_TYPE_ARRAY`, `BPF_MAP_TYPE_PERCPU_HASH`, `BPF_MAP_TYPE_LRU_HASH`, `BPF_MAP_TYPE_RINGBUF`, `BPF_MAP_TYPE_PERF_EVENT_ARRAY`, and a few dozen more specialised ones. - `key_size` and `value_size` in bytes. - `max_entries` — the capacity. - flags. All of these are fixed for the life of the map. Keys are exactly `key_size` bytes; there is no variable-length key and no automatic growth when the map fills. In practice you rarely call the syscall by hand — you declare the map in the BPF C source and the loader creates it: ```c struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 10240); __type(key, __u32); __type(value, __u64); } bytes_by_pid SEC(".maps"); ``` The loader creates the map, gets a file descriptor, and patches that descriptor into the program's instructions before loading it, so the verifier knows exactly which map each helper call touches and can check that the key and value sizes the program uses match the map's declaration. ## The kernel side: helpers only BPF code cannot reach into the map's internals. It calls helpers: - `bpf_map_lookup_elem(&map, &key)` returns a pointer to the value **inside the map**, or NULL. The verifier forces a NULL check before you dereference it; because the pointer is into the map itself, writing through it updates the stored value directly. - `bpf_map_update_elem(&map, &key, &value, flags)` inserts or overwrites. Flags are `BPF_ANY`, `BPF_NOEXIST` (fail if present) and `BPF_EXIST` (fail if absent). - `bpf_map_delete_elem(&map, &key)` removes an entry. All of these return an integer status that a careless program ignores — a real source of silent data loss when a map is full. ## The user-space side: it is a file descriptor User space never sees a kernel pointer. It holds a file descriptor and issues `bpf()` commands against it: `BPF_MAP_LOOKUP_ELEM`, `BPF_MAP_UPDATE_ELEM`, `BPF_MAP_DELETE_ELEM`, and `BPF_MAP_GET_NEXT_KEY` for iteration (you walk a hash map by asking for the key after the one you have; passing a NULL/absent key gets the first). Newer kernels add batch commands so you can drain many entries per syscall. libbpf wraps these as `bpf_map_lookup_elem(fd, &key, &value)` and friends, and `bpf_map__fd(skel->maps.bytes_by_pid)` gets the descriptor from a generated skeleton. The important property: **each of these calls copies**. The key goes into the kernel, the value comes back out, one syscall at a time. Polling a large map every second is real overhead, which is why the aggregation belongs in the BPF program and only the summary belongs in the map. The one genuine exception is `BPF_MAP_TYPE_RINGBUF`, whose buffer user space memory-maps and consumes without a syscall per record. ## Lifetime Because the handle is a file descriptor, a map is reference-counted like any other kernel object. References come from open descriptors, from loaded programs that use it, and from bpffs pins. When the last one goes, the map and its contents are freed. That is why a tool that loads a program and exits leaves nothing behind — and why long-lived or shared maps are pinned into the BPF filesystem so a separate process can reopen them later. ## What people get wrong They assume the map is shared memory both sides poke at directly; it is not, outside of the ring buffer. They assume a map grows when it fills; it does not. And they assume the data outlives the loader by default; it does not.

  • If maps are freed when the last reference goes, how would you keep one alive after the loading process exits?
    Pin it into the BPF filesystem, normally mounted at /sys/fs/bpf. A pin is an extra reference held by a filesystem entry, so the map survives the loader's exit and any other privileged process can reopen it by path and keep reading or writing it. Removing the pin drops that reference again.
  • How do you iterate over every entry of a BPF hash map from user space, and what should you be careful about?
    Repeatedly issue BPF_MAP_GET_NEXT_KEY, feeding back the key you just received, and look up each key as you go. It is not a snapshot: the BPF program keeps inserting and deleting while you walk, so you can miss entries or see ones that vanish before the lookup. Batch commands cut the syscall count but not the raciness.
  • Why is it better to aggregate inside the BPF program than to export every event and aggregate in user space?
    Every record you export costs a copy and, for per-key polling, a syscall; a busy hook can fire millions of times a second, so the export path becomes the bottleneck and starts dropping data. Counting or histogramming in the kernel turns millions of events into a handful of map entries that user space reads cheaply.

  • Thinks a map is shared memory both sides write to directly
  • Says the map grows automatically when max_entries is reached
  • Assumes map data survives the loader process by default
  • Believes BPF programs can write into user-space process memory
  • Thinks keys can be any length rather than exactly key_size

context

open as a page

An eBPF program is loaded into the Linux kernel with a fixed program type such as BPF_PROG_TYPE_KPROBE or BPF_PROG_TYPE_XDP. What does that program type decide about the program, and why can't you attach one program anywhere you like?

level: juniorimportance: must knowfreq 72%

basics

~20 s

An eBPF program's type fixes which kernel hooks it may attach to, the layout of the single context argument it receives, which BPF helper functions it may call, and how the kernel interprets its return value.

open as a page

In Linux eBPF, at what point does the kernel's verifier examine your program, and what happens to a program that fails verification?

level: juniorimportance: must knowfreq 70%

basics

~20 s

The eBPF verifier runs inside the kernel at load time, when the bpf() syscall submits the program — before it is attached to anything and before a single instruction executes. A rejected program never runs: the load call fails and the kernel returns a verifier log saying why.

open as a page

On a Linux server with two blank disks, explain LVM's three layers — physical volume, volume group and logical volume — and give the commands, in order, that turn those disks into a mounted filesystem.

level: juniorimportance: must knowfreq 78%

basics

~20 s

LVM stacks three layers: pvcreate initialises each disk as a physical volume, vgcreate pools those PVs into a volume group, and lvcreate carves logical volumes out of the pool. Then you mkfs and mount the LV like any block device.

open as a page

On a Linux host using LVM, you run `lvextend -L +50G /dev/vg0/data` and it reports success, yet `df -h` still shows the old size for the mounted filesystem. Why, and what do you run to make the filesystem use the new space?

level: juniorimportance: must knowfreq 76%

basics

~20 s

An LVM logical volume and the filesystem on it are separate layers. lvextend grew only the block device, so the filesystem still records its old size. Run resize2fs (ext4) or xfs_growfs (XFS) afterwards, or pass lvextend -r to do both.

open as a page

A packaged Linux binary exits immediately with a vague message and writes no log. How do you use strace to find out which file or permission it is actually failing on?

level: juniorimportance: must knowfreq 70%

basics

~20 s

Run strace -f -e trace=file on the program and read the failing calls near the end: openat returning -1 ENOENT names a missing path, EACCES names a permission problem. The failed system call is the real error.

open as a page

An eBPF tool needs to stream individual event records to a user-space agent. What does BPF_MAP_TYPE_RINGBUF give you that BPF_MAP_TYPE_PERF_EVENT_ARRAY does not, and when would you still reach for the perf buffer?

level: middleimportance: must knowfreq 66%

basics

~20 s

BPF_MAP_TYPE_RINGBUF is one buffer shared by all CPUs: events keep their global order, memory does not scale with CPU count, and bpf_ringbuf_reserve() lets a program fill a record in place. The perf buffer is per-CPU and is the only option below Linux 5.8.

open as a page

In Linux eBPF networking, what does a program attached to the tc (traffic control) hook see that an XDP program does not, and when would you choose tc over XDP?

level: middleimportance: must knowfreq 55%

basics

~20 s

A tc BPF program runs after the kernel has built an sk_buff, so it sees packet metadata, works on every driver, and can be attached on egress as well as ingress. XDP is faster but ingress-only and metadata-free.

open as a page

In Linux XDP (eXpress Data Path), where in the packet receive path does an XDP program run, and what do the return codes XDP_PASS, XDP_DROP, XDP_TX, XDP_REDIRECT and XDP_ABORTED each tell the driver to do?

level: middleimportance: must knowfreq 62%

basics

~20 s

An XDP program runs inside the NIC driver's receive routine, before the kernel allocates an sk_buff. Its return value is a verdict: PASS to the normal stack, DROP the packet, TX back out the same NIC, REDIRECT elsewhere, ABORTED for error drops.

open as a page

How does the Linux kernel physically attach an eBPF program at a kprobe compared with at a tracepoint, and what does that mechanical difference mean for what you can attach to?

level: middleimportance: must knowfreq 62%

basics

~20 s

A kprobe patches the live kernel, replacing the instruction at a symbol's address with a trap or a jump, so it can go almost anywhere. A tracepoint is static instrumentation compiled into the source and exists only where a kernel developer placed one.

open as a page

Two ways to build an eBPF tool on Linux are the bcc framework and libbpf with CO-RE. Where does each one compile the BPF C program, and what does that difference cost you when you ship the tool to a fleet of servers?

level: middleimportance: must knowfreq 62%

basics

~20 s

bcc compiles the BPF C source with an embedded Clang on every host at run time, so LLVM and matching kernel headers must be installed there. libbpf with CO-RE compiles once into a portable binary and relocates struct field offsets at load time using the kernel's own BTF.

open as a page

You need a latency distribution for block I/O on a busy production server, over ten minutes at tens of thousands of I/Os per second. Explain how an eBPF tracing tool such as bpftrace or bcc's biolatency builds that histogram without shipping one record per I/O to user space, and why that design is what makes it safe to run on a live box.

level: middleimportance: must knowfreq 45%

basics

~20 s

The BPF program does the maths in the kernel. It stores a start timestamp per in-flight request, computes the delta on completion, and increments a bucket in a kernel-resident map. User space reads that map once at the end, so the cost per event is a few lookups rather than a record pushed through a ring buffer.

open as a page

An eBPF program calls bpf_map_lookup_elem() and dereferences the returned pointer straight away, and the kernel refuses to load it with an invalid memory access mentioning map_value_or_null. What is the verifier objecting to, and what does it want the code to do?

level: middleimportance: must knowfreq 62%

basics

~20 s

bpf_map_lookup_elem() returns NULL when the key is absent, so the verifier types its result as a maybe-NULL pointer and refuses any dereference of it. An explicit NULL check on the returned pointer refines the type on the surviving branch, after which the dereference is allowed.

open as a page

A team asks you to shrink an LVM logical volume holding a 500 GB filesystem down to 200 GB so the freed extents can go to another volume. What do you check first, and what is the correct order of operations?

level: middleimportance: must knowfreq 61%

basics

~20 s

Check the filesystem type first. XFS cannot shrink at all, so the only route is back up, recreate a smaller volume and restore. On ext4, unmount, shrink the filesystem with resize2fs, and only then reduce the volume with lvreduce — never the other way round.

open as a page

You run `lvcreate -s -L 5G -n db-snap vg0/db` on a Linux host. What has LVM created, and what happens on disk the first time a block of the origin volume is overwritten afterwards?

level: middleimportance: must knowfreq 62%

basics

~20 s

LVM allocated a 5 GiB copy-on-write area and a snapshot device; no data was copied. The first overwrite of a chunk on the origin makes device-mapper read the old chunk, write it into the copy-on-write area, record the exception, then apply the new write.

open as a page

A libbpf-based tool loads and attaches an eBPF program on a Linux host, then the user-space process exits and the program vanishes from `bpftool prog show`. What owns a loaded BPF object's lifetime, and how do you make one outlive the process that loaded it?

level: seniorimportance: must knowfreq 44%

basics

~20 s

Loaded BPF programs, maps and links are refcounted kernel objects with no name in any namespace; the loader's file descriptors are usually the only reference, so closing them at process exit frees everything. Pinning to the bpffs filesystem at /sys/fs/bpf takes an extra reference that survives the process.

open as a page

A nightly job snapshots a busy Linux volume so a backup can stream from it. This morning `lvs` shows the snapshot at 100% Data% with an `I` in its attribute string, and the backup that ran from it is unusable. What happened, and how do you size and monitor the snapshot so it does not recur?

level: seniorimportance: must knowfreq 55%

basics

~20 s

The origin changed more than the snapshot's copy-on-write area could hold, so LVM marked the snapshot invalid and reads from it now fail. Size the COW area for the origin's write volume during the snapshot's lifetime, monitor Data%, and enable snapshot autoextend with lvm2-monitor running.

open as a page

Walk through turning a `perf record` capture of a hot Linux service into a flame graph, and explain how to read the result — what the width of a box means, and what the horizontal axis does not mean.

level: seniorimportance: must knowfreq 58%

basics

~20 s

Record with call graphs, dump the samples as text with perf script, fold each stack to one line with stackcollapse-perf.pl, then render with flamegraph.pl. Box width is the share of samples containing that frame; the x axis is only alphabetical ordering, never time.

open as a page

Why can attaching strace to a busy production process slow it down by an order of magnitude, and what would you do instead when you cannot afford that?

level: seniorimportance: must knowfreq 52%

basics

~20 s

strace uses ptrace, which stops the traced process twice for every system call and wakes strace to inspect it. Each traced call becomes several context switches, so a syscall-heavy process can slow by ten to a hundred times.

open as a page

You are handed a Linux box with bpftrace installed and asked to watch which files a service opens. Walk through what the one-liner `bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'` actually does, and explain how you would have discovered that probe and its argument fields in the first place.

level: juniorimportance: should knowfreq 40%

basics

~20 s

The one-liner attaches a small BPF program to the openat syscall-entry tracepoint and prints the calling process name and the filename for every openat on the whole machine. bpftrace -l lists available probes and bpftrace -lv shows a tracepoint's argument fields.

open as a page

A colleague says the nightly LVM snapshot of the server's data volume is the backup. Why is an LVM snapshot sitting in the same volume group not a backup, and what is it actually good for?

level: juniorimportance: should knowfreq 48%

basics

~20 s

An LVM snapshot stores only the blocks that changed since it was taken and reads everything else from the origin volume, on the same disks in the same volume group. Lose the origin or the disks and the snapshot is worthless. It is a stable point-in-time source to copy from, not a copy.

open as a page

A Linux box is pinned at 100% CPU and you have already identified the guilty process. What does `perf top` show you that a process-level view cannot, and how does it collect that information?

level: juniorimportance: should knowfreq 52%

basics

~20 s

perf top is a live sampling profiler. It interrupts the CPU many times a second, records which instruction was executing, resolves it to a function, and ranks functions by how many samples landed in them — so it names the hot code, not just the hot process.

open as a page

In eBPF, what is a helper function, and why can a BPF program call bpf_map_lookup_elem() but not an arbitrary kernel function such as kmalloc()?

level: middleimportance: should knowfreq 55%

basics

~20 s

BPF helpers are a fixed set of kernel functions, each with a numbered ID and a prototype the verifier knows, that BPF programs are allowed to call. Arbitrary kernel functions like kmalloc() have no such contract and cannot be proven safe.

open as a page

You have SSH'd into a Linux host you did not set up and want to know which eBPF programs are loaded and what they are attached to. Which bpftool subcommands answer that, and what do their outputs tell you?

level: middleimportance: should knowfreq 45%

basics

~20 s

Start with bpftool prog show for loaded programs and bpftool map show for their maps. Attachment points come from bpftool link show, bpftool net show for interface hooks and bpftool cgroup tree for cgroup hooks, because prog show alone does not say where a program is attached.

open as a page

A bpftrace script that attaches with a `kprobe:` on a kernel function stops attaching after your fleet is upgraded to a newer kernel, while a colleague's script using a `tracepoint:` probe for roughly the same event keeps working. Why do the two behave differently across kernel versions, and what does that mean for a tracing tool you have to ship to many machines?

level: middleimportance: should knowfreq 36%

basics

~20 s

A kprobe attaches to an internal kernel function name with no stability promise, so a rename, an inline or a signature change breaks it on the next kernel. A tracepoint is a declared instrumentation point with named fields that kernel developers try not to break, so it survives upgrades far better.

open as a page

Older Linux kernels rejected any eBPF program containing a backward jump. What does the verifier require of a loop today, and what problem does the bpf_loop() helper solve?

level: middleimportance: should knowfreq 45%

basics

~20 s

The verifier must prove every eBPF program terminates. Originally it demanded a loop-free control-flow graph, so loops had to be unrolled; since Linux 5.3 it accepts loops whose iteration count it can bound, and bpf_loop() moves the iteration to runtime so verification cost stops scaling with the trip count.

open as a page

The verifier rejects an eBPF program because its stack usage is too large. How much stack does an eBPF program get on Linux, why is the limit so small, and where should a large scratch buffer live instead?

level: middleimportance: should knowfreq 40%

basics

~20 s

An eBPF program gets 512 bytes of stack, and with subprograms it is the combined depth of the call chain that must fit. The limit is small because the program runs on the ordinary kernel stack in any context. Large buffers belong in a per-CPU array map used as scratch space.

open as a page

When creating an LVM logical volume with lvcreate, what is the difference between the default linear layout and a striped one requested with -i, and when does striping actually buy you anything?

level: middleimportance: should knowfreq 42%

basics

~20 s

A linear LV concatenates extents, filling one physical volume before using the next, so one request hits one disk. A striped LV created with lvcreate -i alternates fixed-size chunks across several PVs, so a large sequential transfer is served by all of them in parallel.

open as a page

In LVM on Linux, what is a physical extent (PE), what extent size does vgcreate use by default, and what does that choice actually affect?

level: middleimportance: should knowfreq 52%

basics

~20 s

A physical extent is LVM's fixed-size unit of allocation — 4 MiB by default in LVM2, set per volume group at vgcreate time and shared by every PV in it. Logical volume sizes are always rounded up to a whole number of extents.

open as a page

An LVM volume group on a Linux server has no free extents left. In one case the storage team presents a brand-new disk as /dev/sdc; in another they enlarge the existing disk that the volume group already sits on. What do you run in each case to get that capacity into an existing logical volume?

level: middleimportance: should knowfreq 54%

basics

~20 s

A new disk is initialised with pvcreate and joined to the group with vgextend. A disk enlarged in place needs a device rescan and then pvresize on the existing physical volume. In both cases lvextend -r is what finally moves the space into a volume.

open as a page

showing 1–30 of 60