A `perf record -g` capture of a production Linux service produces a report where nearly every call stack is only one or two frames deep. Why does frame-pointer unwinding fail, and what do `--call-graph dwarf` and `--call-graph lbr` do differently?

## What a call graph costs A sample without a stack tells you which function was executing. A sample *with* a stack tells you who called it, which is usually the actionable half — `memcpy` being hot means nothing until you know which code path is calling it. Getting that stack means unwinding from the sampled instruction pointer back through the caller chain, and there are exactly three mechanisms available, each with a different bargain. ## Frame pointers: `--call-graph fp` (what `-g` means) A frame pointer is a register holding the base of the current stack frame, with each frame storing the caller's value. Unwinding is then trivial: follow the linked list. It is cheap enough to do in the kernel during the sample itself. The catch is that on x86-64 the frame pointer register is a perfectly good general-purpose register, and compilers have long defaulted to `-fomit-frame-pointer` at optimisation levels above `-O0`. When the register is repurposed there is no chain, and the unwind terminates almost immediately. That is exactly the symptom in the question: one or two frames, then nothing. It gets worse in mixed builds. Your application may have frame pointers while the system libraries it calls do not, so stacks truncate the moment they pass into `libc` or a third-party shared object — a partial, misleading picture that is arguably more dangerous than none. **The fix at the source:** rebuild with `-fno-omit-frame-pointer`. The runtime cost is small — one register and a couple of instructions per call — and both Fedora (from Fedora 38) and Ubuntu (from 24.04 LTS) now build their distribution packages this way precisely so that production profiling works. On your own service, adding the flag to release builds is usually the single highest-leverage change for profilability. ## DWARF: `--call-graph dwarf` When you cannot rebuild, DWARF unwinding works on binaries that have no frame pointers at all. The mechanism is blunt: with each sample perf copies a chunk of the user stack — 8 KB by default — into the sample record along with the register state. Nothing is unwound at sample time. Later, `perf report` or `perf script` replays that stack memory against the DWARF unwind information for the binaries involved and reconstructs the frames. The consequences follow from the mechanism: - **`perf.data` gets very large.** 8 KB per sample at hundreds of samples a second per CPU adds up fast. Drop `-F` and shorten the capture. - **Deep stacks get truncated.** If the live stack is deeper than the copied window, the tail is missing. `--call-graph dwarf,16384` or larger buys depth at the cost of even more data. - **Debug information must be available** at analysis time — the `-dbgsym` package on Debian/Ubuntu, `dnf debuginfo-install` on Fedora/RHEL — and it must match the exact build. - **Overhead is real** at capture time too, because copying stack memory on every sample is not free. ```bash perf record --call-graph dwarf,16384 -F 49 -p 4242 -- sleep 30 perf report --stdio --no-children ``` ## LBR: `--call-graph lbr` Intel CPUs maintain last-branch records, a small hardware ring of recent branch source/target pairs. perf can reconstruct a call chain from them essentially for free — no stack copying, no frame pointers, no debug data. The limits are hardware limits. The ring is shallow (a couple of dozen entries depending on the microarchitecture), so deep stacks are cut off; it is user-space only, so kernel frames are absent; and it is Intel-specific, which matters on a mixed or Arm fleet. For a shallow, hot user-space call path it is the cheapest good answer available. ## Reading the report once stacks work One more thing separates people who have used this from people who have read about it: `perf report` defaults to `--children`, which shows *cumulative* overhead — time in a function plus everything it called. That is what you want for finding an expensive subtree. `--no-children` shows *self* overhead, the time in the function's own instructions, which is what you want for finding the actual hot loop. Misreading one for the other sends people optimising a wrapper that costs nothing itself. ## Choosing, in practice 1. Can you rebuild the hot code? `-fno-omit-frame-pointer` and plain `-g`. Best signal, lowest capture cost, works for kernel frames too. 2. Production binary you cannot touch, and debug data available? `--call-graph dwarf` with a reduced sample rate and a short window. 3. Intel host, shallow user-space stack, minimal overhead required? `--call-graph lbr`. And always sanity-check the first report: if the deepest stacks are two frames, stop and fix the unwinding before you draw any conclusion from the data. A truncated profile does not fail loudly — it quietly attributes everything to the wrong place.