Tracing and Observability

## The design that does not work The obvious way to measure latency is to emit a record per event and do the arithmetic later: timestamp, device, duration, one line each. At tens of thousands of events per second this fails on three counts. Each record must be copied out of the kernel; a user-space reader must be scheduled to drain it; and something must store or parse the stream. The measurement starts to cost more than the work being measured, and worse, the cost lands on the very machine that is already in trouble. This is the structural reason older per-event tracing approaches were reserved for a lab. ## What eBPF does instead The defining property of eBPF observability is that **the reduction happens in the kernel, at the event, before anything is copied out**. A histogram tool is built from two probe blocks plus a map: 1. **Start probe.** On the issue path, record the current time: `@start[key] = nsecs;`. The key identifies the in-flight operation — a thread id for a syscall, a device plus sector for a block request. 2. **End probe.** On the completion path, look the key up; if present, compute `nsecs - @start[key]`, feed it into the histogram, and delete the entry so the map does not grow without bound. In bpftrace that is roughly: ``` kprobe:vfs_read { @start[tid] = nsecs; } kretprobe:vfs_read /@start[tid]/ { @us = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]); } ``` `hist()` is not a function that returns data to you — it is an *aggregation* that lives in a kernel map. Each call computes which power-of-two bucket the value falls into and increments that bucket's counter. bcc's `biolatency` does exactly the same thing against the block layer. ## Why this is cheap Per event you pay: one map insert, one map lookup, one delete, a subtraction, and a bucket increment. That is a small number of hash operations, typically well under a microsecond, executed inline on the thread that was going to run anyway. Nothing is written to a pipe, nothing wakes a user-space process, no context switch occurs. The amount of data that ever crosses into user space is bounded by the *shape* of the aggregation, not by the event rate. A log2 latency histogram has a few dozen buckets whether it saw a thousand events or a billion. Ten minutes at 30,000 events per second is 18 million samples and still one small map read at the end. A second reason it is safe: the program was verified before it loaded. It cannot loop unboundedly, cannot dereference an unchecked pointer, and cannot be attached in a way that lets it panic the kernel. The failure mode of a bad tracing program is a load-time rejection, not a crashed production host. ## The counters that must be maintained The start map is the part that bites people. If the completion probe never fires for some requests — the operation errors out on a path you did not instrument, or the tool starts mid-flight — entries accumulate. BPF maps have a fixed maximum number of entries fixed at load time, so a leaking start map eventually stops accepting inserts and you silently under-count. Always delete on the completion path, and be suspicious of a histogram whose sample count is far below the event rate you expect. ## When you genuinely need per-event output Aggregation answers "what is the distribution"; it cannot answer "which request was the slow one, and what was running at the time". For that you do stream events — bcc's `biosnoop` prints one line per I/O with the process, device, sector and latency, and bpftrace's `printf()` does the same. These push records through a per-CPU ring buffer that user space drains. The buffer is finite, so under load records are dropped and the tool tells you it lost events. That is an acceptable trade for a short, targeted window or a low-rate event (process execs, for instance), and a bad one for every block I/O on a busy database host. The practical workflow is therefore two-stage: aggregate first to find out *whether* there is a tail and where it sits, then stream a narrow, filtered slice — one device, one process, one latency threshold — to find out *what* is in it. Filtering in the probe's predicate keeps the second stage cheap, because the discard also happens in the kernel. ## What to say about overhead Be concrete rather than absolute. Overhead is roughly the per-event cost times the event rate, so the question is always "how hot is this probe". Attaching to something that fires a million times a second is expensive even at a microsecond each; attaching to process exec is free in practice. Tracing tools that let you filter early and aggregate in the kernel let you keep that product small, which is the actual skill.