Two ways to build an eBPF tool on Linux are the bcc framework and libbpf with CO-RE. Where does each one compile the BPF C program, and what does that difference cost you when you ship the tool to a fleet of servers?
answer
- where the compiler runs
- one needs headers, one needs BTF
- runtime Clang versus load-time relocation
- offsets patched against /sys/kernel/btf/vmlinux
- ship a binary, not a toolchain
basics
~20 sbcc compiles the BPF C source with an embedded Clang on every host at run time, so LLVM and matching kernel headers must be installed there. libbpf with CO-RE compiles once into a portable binary and relocates struct field offsets at load time using the kernel's own BTF.
solid answer
~50 sbcc embeds LLVM/Clang in the tool itself: the BPF C program is a string in your Python or C++ source, and it is compiled **on the target machine, every time the tool starts**. That means each production host needs Clang plus kernel headers that match the running kernel, startup costs seconds, and the process briefly takes hundreds of megabytes of RSS for the compiler. libbpf takes the opposite approach — you compile the BPF C once with `clang -g -O2 -target bpf`, which emits a `.o` carrying BTF type information and CO-RE relocation records. At load time libbpf reads the running kernel's BTF from `/sys/kernel/btf/vmlinux` and patches struct field offsets in the instruction stream to match that kernel. The result is one small static binary you can copy anywhere, with no toolchain on the box. The price is that the target kernel must expose BTF, which bcc does not require.
code
bash · 3 linesclang -g -O2 -target bpf -D__TARGET_ARCH_x86 \
-c count_exec.bpf.c -o count_exec.bpf.o
bpftool gen skeleton count_exec.bpf.o > count_exec.skel.hgo deeper
Know that eBPF programs are written in restricted C and must get into the kernel somehow, and be able to say that bcc compiles that C on the machine while libbpf compiles it ahead of time.
Explain the mechanics both ways: bcc's embedded Clang plus kernel headers versus a -g compile that emits BTF and relocation records which libbpf patches against the running kernel at load.
Argue the operational case — no compiler or headers on production hosts, millisecond startup, one binary to distribute — and name the one thing CO-RE requires in exchange, kernel BTF, plus what you do on hosts that lack it.
Own the standard: which model your organisation ships, the minimum kernel and BTF baseline that implies, and how you keep an escape hatch for hosts that fall outside it without maintaining two codebases forever.
## The problem both approaches are solving A BPF program that reads kernel data structures is compiled against a particular kernel's headers. Field offsets inside structures like `task_struct` move between kernel versions and even between distro configurations, because a `CONFIG_` option that adds one member shifts everything after it. A binary compiled on one kernel and run on another would silently read the wrong bytes. bcc and libbpf answer this in two completely different ways. ## bcc: compile on the target, every time In bcc the BPF C program is data — a string in the Python source, or a file read at startup. The `BPF()` constructor hands that string to an LLVM/Clang instance linked into libbcc, which compiles it against the headers of the *currently running* kernel and then loads the result via the `bpf()` syscall. ```python from bcc import BPF b = BPF(text="int hello(void *ctx) { return 0; }") ``` Because compilation happens on the box, offsets are correct by construction. That is genuinely elegant for exploratory work, and it is why bcc dominated the early tracing-tool era. The costs land on operations: - **Toolchain on production hosts.** libbcc drags in LLVM and Clang — a large dependency for a machine that is supposed to run one application. - **Kernel headers must be present and must match.** They come from the distro's kernel-headers/`kernel-devel` package under `/lib/modules/$(uname -r)/build`, or from the `kheaders` module, which exposes them as `/sys/kernel/kheaders.tar.xz`. On a minimal or immutable image they are usually absent, and after a kernel upgrade with no reboot they are the *wrong* ones. - **Startup latency and memory.** Compiling takes seconds and the compiler's working set is large, which is unpleasant when you are trying to catch a short-lived incident and awful when the tool runs on thousands of hosts. - **Failure mode is a build error at 3am**, in a language most on-call engineers will not debug under pressure. ## libbpf and CO-RE: compile once, relocate at load CO-RE means "Compile Once — Run Everywhere". You build the BPF object ahead of time: ```bash clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -c probe.bpf.c -o probe.bpf.o ``` `-g` is what matters: it makes Clang emit BTF, a compact type-description format, into the object. When the source accesses a kernel struct field through libbpf's helpers — `BPF_CORE_READ()`, or the `__builtin_preserve_access_index` machinery behind it — the compiler also records a *relocation*: "this instruction loads field `pid` of `struct task_struct`; patch its offset at load time". At run time libbpf reads the running kernel's own BTF from `/sys/kernel/btf/vmlinux`, matches each recorded type and field against it, and rewrites the offsets in the instruction stream before handing the program to the verifier. The same binary therefore loads on kernels whose layouts differ, without a compiler ever touching the host. You typically ship a single statically linked executable with the BPF object embedded in it via a generated skeleton header. ## The trade you are actually making | | bcc | libbpf + CO-RE | |---|---|---| | Compiler on target | required | not required | | Kernel headers on target | required | not required | | Kernel BTF on target | not required | required | | Startup | seconds | milliseconds | | Distribution unit | source + runtime + LLVM | one binary | The single thing CO-RE demands that bcc does not is kernel BTF, which exists only when the kernel was built with `CONFIG_DEBUG_INFO_BTF=y`. Mainstream distro kernels have shipped it since roughly 2020, but very old or custom-built kernels may not, and then a CO-RE binary cannot resolve its relocations. ## Where each still belongs Use bcc's Python API for one-off exploration on a lab box where a toolchain is fine and iteration speed is everything. Use libbpf CO-RE for anything you *ship*: agents, always-on collectors, tools handed to other teams. The bcc project itself acknowledges this — its `libbpf-tools` directory contains CO-RE rewrites of the classic tools precisely so they can be distributed as plain binaries.
- What does the -g flag actually add to the compiled BPF object, and why does CO-RE stop working without it?`-g` makes Clang emit BTF type information plus the CO-RE relocation records into the `.o`. Those records are what libbpf consults at load time to decide which instructions carry a struct field offset that needs patching. Without `-g` the object has no BTF, libbpf has nothing to relocate against the kernel's types, and any access to a kernel structure keeps the build machine's hardcoded offsets.
- Does CO-RE protect you when a kernel renames or removes a struct field entirely, not just moves it?No. Relocation adjusts offsets for fields that still exist; it cannot invent a field that was removed or renamed. libbpf will fail the relocation and the load. You handle that in source with `bpf_core_field_exists()` and take a different code path, or read an alternative field, so one binary can cope with both layouts.
- Why do bcc-based tools sometimes fail on a host that has just been kernel-upgraded but not rebooted?bcc compiles against the headers it finds for the running kernel. After a package upgrade the headers on disk describe the *new* kernel while the *old* one is still running, so struct layouts can disagree — the tool either fails to build or, worse, reads fields at the wrong offsets. A CO-RE binary is immune because it relocates against the live kernel's BTF.
saying these in an interview costs you the question
- Saying bcc ships precompiled BPF bytecode like libbpf does
- Claiming CO-RE recompiles the program on each host
- Thinking kernel headers are needed for a CO-RE binary
- Assuming BTF is present on every Linux kernel
- Believing bcc's runtime compile is free because it is cached