You want to time a function inside a running user-space application using bpftrace's `uprobe` and `uretprobe` probes. How does a uprobe actually attach to the process, roughly what does each hit cost compared with a kernel tracepoint, and which kinds of binary make this approach unreliable?

## How the attachment works A uprobe is dynamic instrumentation of user space, driven from the kernel. You name a **file and a symbol**, and the kernel resolves the symbol to an offset within that file. It then arranges that, wherever that file is mapped executable, the instruction at that offset is replaced by a breakpoint. The replacement happens in a **private, copy-on-write copy of the page**, not in the file on disk and not in the page cache copy other processes share. So instrumenting `malloc` in libc does not corrupt the shared library or affect processes you did not target when you have scoped the probe. When a thread executes the patched instruction it traps into the kernel, the kernel runs your BPF program, then executes the displaced original instruction out of line and returns control. Because the attachment is keyed to the *file*, a uprobe on a library applies to every process that maps it unless you filter — hence the near-universal `/pid == $1/` predicate in real scripts. ## What it costs Every hit is a trap: a full transition into the kernel and back, plus the out-of-line single step. That puts a uprobe in the region of a microsecond per hit, versus a tracepoint or an optimised kernel probe measured in tens or low hundreds of nanoseconds. Concretely, instrumenting a function called ten thousand times a second is unremarkable; instrumenting one called ten million times a second will visibly slow the application. The cost is per *hit*, so scoping to one process reduces the event rate but not the per-event price. The practical discipline is the same as for kernel tracing: filter in the predicate so the expensive work is skipped, and aggregate rather than print. ## Getting a symbol to attach to The symbol must be findable in the object file. Three situations recur: **Dynamically linked programs.** The code for a library function lives in the library, so you attach to the `.so`, not the executable: ``` bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:getaddrinfo { printf("%s %s\n", comm, str(arg0)); }' ``` Attaching to the executable for a libc function finds nothing — a very common first mistake. `ldd` on the binary, or the maps of the running process, tell you which object is actually loaded. **Statically linked programs.** Everything is in the one binary, so you attach there — but only if the symbols survived. **Stripped binaries.** With no symbol table there is nothing to resolve. You can still attach if you know the *offset*, which you obtain from a copy of the binary that has symbols or from separate debug information. bpftrace accepts an address form for exactly this case. Distribution debuginfo packages are the usual source. You can enumerate what is attachable with a listing, e.g. `bpftrace -l 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:*'`, and the classic binutils tools (`nm -D`, `readelf -s`, `objdump`) show the symbol table directly. ## Where uretprobes go wrong A `uretprobe` cannot simply patch the function's return instruction — a function may have many exits and may be entered from anywhere. Instead the kernel patches the *entry*, saves the caller's return address, and substitutes an address of its own so that when the function returns, control lands back in the kernel, which then fires the probe and jumps to the real return address. This is a stack manipulation, and it breaks when the runtime has its own opinions about the stack: - **Go.** Goroutine stacks start small and are moved when they grow, and Go's calling convention has not matched the C one that bpftrace's positional `arg0`, `arg1` built-ins assume. Return probes on Go binaries are a well-known source of wrong results and crashes; entry probes with hand-decoded arguments are more workable, but Go tracing is genuinely specialist work. - **Runtimes that do their own unwinding or non-local exits.** Anything that longjmps past the instrumented return, or unwinds through it for exception handling, can leave the substituted return address unused. - **JIT-compiled code.** Java, Node and similar runtimes generate machine code at runtime. There is no file offset and no symbol in an ELF object to attach to at all, so uprobes are the wrong instrument entirely. ## The stable alternative: USDT USDT (userland statically defined tracing) probes are markers the application's own authors compiled in — a name, a provider, and a defined set of arguments, recorded in an ELF note. Attaching to one costs about the same as a uprobe, but you get an interface the application maintainer intends to keep, at a semantically meaningful point, rather than a symbol that the next release may inline away. glibc ships USDT markers, as do several language runtimes when built with the option enabled. bpftrace attaches to them with `usdt:/path/to/binary:provider:name` and lists them with `bpftrace -l 'usdt:/path/to/binary:*'`. If the application you are tracing has them, prefer them. ## The practical checklist Confirm which object holds the code; confirm the symbol exists there; filter by pid; prefer entry probes and aggregation over return probes and per-event printing; and when the target is Go or a JIT runtime, reach for the runtime's own instrumentation before reaching for uprobes.