A Linux box is pinned at 100% CPU and you have already identified the guilty process. What does `perf top` show you that a process-level view cannot, and how does it collect that information?

## Two different questions about a busy CPU Process-level tools answer an accounting question: over the last interval, how much CPU time did each *process* consume? That is enough to find the offender and nowhere near enough to fix it. `perf top` answers the next question down: inside that CPU time, which *functions* were actually executing? ## How sampling works `perf top` is a sampling profiler. It opens a performance event through the kernel's perf subsystem and asks to be notified periodically. The default event is the hardware `cycles` counter from the CPU's performance monitoring unit (PMU); on hosts where no PMU is exposed — many virtual machines — perf falls back to the software `cpu-clock` event, which is driven by a timer instead. Each time the counter overflows, the kernel takes a **sample**: the instruction pointer at the moment of the interrupt, the PID and TID, the CPU, and — if you passed `-g` — a call chain walked back from that point. Crucially this is *not* instrumentation. The target is not recompiled, not restarted, not wrapped in anything. You attach to a process that is already misbehaving in production, which is exactly the situation the question describes. Because it samples, every number you read is statistical. A symbol credited with 40% of the profile is one in which 40% of the *collected samples* landed. With a few thousand samples that is a reliable estimate of where CPU time goes; with fifty samples it is noise. This is the single most common misreading of profiler output. ## Reading the output The live table has three columns that matter: - **Overhead** — the share of samples attributed to this symbol. - **Shared Object** — the DSO the sampled address belongs to: the executable itself, a shared library such as `libc.so.6`, or `[kernel.kallsyms]`. - **Symbol** — the function name, prefixed with `[.]` for a user-space address and `[k]` for a kernel one. That prefix is a fast triage signal on its own. A profile dominated by `[k]` frames says the time is going into kernel work — syscalls, memory management, network processing — while a profile dominated by `[.]` frames in your own binary points at application code. ## Narrowing the view ```bash perf top -p 4242 # one process only perf top -e cycles:u # user-space cycles only perf top -F 99 -g # 99 Hz, with call graphs perf top --sort dso,symbol # group by library first ``` The `:u` and `:k` event modifiers restrict sampling to user or kernel mode. Lowering `-F` reduces the profiler's own cost on an already-saturated box. Pressing `a` on a selected symbol drops into annotation, attributing samples to individual instructions inside that function — useful when a single loop is the problem. ## Where it falls down 1. **It is ephemeral.** `perf top` is a live view with no artifact at the end. For anything you want to keep, compare, or hand to someone else, use `perf record` to write a `perf.data` file and analyse it with `perf report`. 2. **Symbols may be missing.** Stripped binaries and packages without debug information produce rows of raw hex addresses or `[unknown]`. The fix is installing the matching debug symbols (`-dbgsym` packages on Debian/Ubuntu, `dnf debuginfo-install` on Fedora/RHEL), not changing anything about the sampling. 3. **Kernel symbols can be hidden.** If `kernel.perf_event_paranoid` or `kernel.kptr_restrict` are restrictive, kernel addresses resolve to zeros and you lose the `[k]` side of the picture. 4. **It only sees on-CPU time.** A process that is slow because it is blocked — waiting on disk, on a lock, on a socket — is not running, so it generates no CPU samples. A quiet profile on a slow service is itself a finding: the problem is not CPU. 5. **The profiler costs something.** At high sample rates on a busy machine, the sampling interrupts themselves consume CPU; the kernel also throttles beyond `kernel.perf_event_max_sample_rate`. ## What a good answer sounds like "I'd attach `perf top -p <pid>` for thirty seconds, look at whether the overhead is concentrated in user or kernel symbols, and if one symbol dominates I'd add `-g` to see who is calling it. Then I'd switch to `perf record` to capture something I can keep." That sequence — live triage, then capture — is what the interviewer is listening for.