How does the Linux kernel physically attach an eBPF program at a kprobe compared with at a tracepoint, and what does that mechanical difference mean for what you can attach to?

## Two ways to get code to run inside a kernel function The eBPF tracing hooks fall into two families that differ not in what they observe but in **how the kernel arranges for your program to run**. Understanding the mechanism is what lets you predict which attachments will fail, which will be stable across kernels, and how you get at the arguments. ## kprobes: patch the running kernel A kprobe is dynamic instrumentation. You give the kernel an address — usually by naming a symbol, optionally with an offset — and the kprobe machinery: 1. Copies away the original instruction at that address. 2. Replaces it with a breakpoint instruction (`int3` on x86-64). 3. When a CPU hits the breakpoint, the trap handler runs the registered probe (your eBPF program), then single-steps or emulates the saved instruction and resumes. Where it can, the kernel *optimises* the probe: instead of a trap it patches in a jump to a generated trampoline, which is considerably cheaper than taking an exception. Either way the mechanism is a live modification of kernel text. The consequences follow directly: - **Coverage is enormous.** Almost any function the compiler actually emitted is probeable. You enumerate candidates with `grep` over `/proc/kallsyms`. - **Some functions are off limits.** Functions marked `notrace` or `noinstr`, and anything on the kernel's kprobe blacklist, cannot be probed — probing them would recurse into the very machinery handling the probe. - **Inlined functions are not there.** If the compiler folded a static function into its callers, there is no symbol and no single instruction stream to patch, and the attach fails with a "no such file or directory" style error even though the function is right there in the source. - **Arguments arrive as registers.** A kprobe program's context is `struct pt_regs`, so libbpf's `PT_REGS_PARM1`/`BPF_KPROBE` macros decode arguments according to the architecture's calling convention. That decoding is only correct if you know the function's real signature, and the signature is an internal kernel detail with no stability promise. - **kretprobe is the return variant.** It works by hijacking the return address, which is why it needs a preallocated pool of in-flight instances. ## Tracepoints: static instrumentation the kernel ships with A tracepoint is a call site placed in the kernel source by a developer using the `TRACE_EVENT` macro family. It is compiled into the kernel image and guarded by a static key (a jump label), so when no one is listening the cost is a nop, and enabling it patches the branch rather than the function. Because it is declared, a tracepoint has metadata. Each one appears under `/sys/kernel/tracing/events/<subsystem>/<name>/` with a `format` file describing its fields and their offsets, and an `id` used to open it as a perf event. Your eBPF program receives a struct matching that layout, with named fields, rather than raw registers. The consequences, again directly: - **You cannot put one where you want it.** Tracepoints exist only where someone placed them. If the code path you care about has none, a kprobe is your only dynamic option. - **The interface is comparatively stable.** Tracepoint fields are a semi-public interface that kernel developers try not to break gratuitously; internal function signatures carry no such expectation. - **The syscall tracepoints are the workhorses.** `syscalls/sys_enter_*` and `syscalls/sys_exit_*` give a well-defined view of syscall entry and exit without probing internal functions at all. There is also a lower-overhead variant: `BPF_PROG_TYPE_RAW_TRACEPOINT`, attached with `BPF_RAW_TRACEPOINT_OPEN`, which hands your program the tracepoint's raw arguments without the perf-format processing in between. ## The attach path itself Historically both families go through the perf subsystem. The loader opens a perf event for the probe — `perf_event_open()` with a kprobe or tracepoint configuration — and then binds the loaded program to it with `ioctl(fd, PERF_EVENT_IOC_SET_BPF, prog_fd)`. Older tooling created kprobes by writing to `kprobe_events` in tracefs first. Modern libbpf prefers a `bpf_link`: `bpf_program__attach_kprobe()` and `bpf_program__attach_tracepoint()` create a link object whose lifetime governs the attachment. ```bash # is there a symbol to probe? sudo grep -w ' do_unlinkat$' /proc/kallsyms # is there a tracepoint, and what fields does it publish? ls /sys/kernel/tracing/events/syscalls/sys_enter_openat/ cat /sys/kernel/tracing/events/syscalls/sys_enter_openat/format ``` ## Choosing between them The rule of thumb an interviewer wants: **prefer a tracepoint when one exists at the place you care about**, because the attachment is cheaper, the fields are named, and it is less likely to evaporate on the next kernel. Reach for a kprobe when no tracepoint covers the path — which is most of the kernel — and accept that you have taken on a dependency on an internal symbol and its calling convention. When you do reach for a dynamic probe on a modern kernel with BTF available, there is a third option built on a different mechanism again, which is worth knowing before you settle on kprobes by default.