Two ways to build an eBPF tool on Linux are the bcc framework and libbpf with CO-RE. Where does each one compile the BPF C program, and what does that difference cost you when you ship the tool to a fleet of servers?

middlemust knowfreq 62%

answer

where the compiler runs
one needs headers, one needs BTF
runtime Clang versus load-time relocation
offsets patched against /sys/kernel/btf/vmlinux
ship a binary, not a toolchain

basics

~20 s

bcc compiles the BPF C source with an embedded Clang on every host at run time, so LLVM and matching kernel headers must be installed there. libbpf with CO-RE compiles once into a portable binary and relocates struct field offsets at load time using the kernel's own BTF.

solid answer

~50 s

bcc embeds LLVM/Clang in the tool itself: the BPF C program is a string in your Python or C++ source, and it is compiled on the target machine, every time the tool starts. That means each production host needs Clang plus kernel headers that match the running kernel, startup costs seconds, and the process briefly takes hundreds of megabytes of RSS for the compiler. libbpf takes the opposite approach — you compile the BPF C once with clang -g -O2 -target bpf, which emits a .o carrying BTF type information and CO-RE relocation records. At load time libbpf reads the running kernel's BTF from /sys/kernel/btf/vmlinux and patches struct field offsets in the instruction stream to match that kernel. The result is one small static binary you can copy anywhere, with no toolchain on the box. The price is that the target kernel must expose BTF, which bcc does not require.

bcc embeds LLVM/Clang in the tool itself: the BPF C program is a string in your Python or C++ source, and it is compiled **on the target machine, every time the tool starts**. That means each production host needs Clang plus kernel headers that match the running kernel, startup costs seconds, and the process briefly takes hundreds of megabytes of RSS for the compiler. libbpf takes the opposite approach — you compile the BPF C once with `clang -g -O2 -target bpf`, which emits a `.o` carrying BTF type information and CO-RE relocation records. At load time libbpf reads the running kernel's BTF from `/sys/kernel/btf/vmlinux` and patches struct field offsets in the instruction stream to match that kernel. The result is one small static binary you can copy anywhere, with no toolchain on the box. The price is that the target kernel must expose BTF, which bcc does not require.

code

bash · 3 lines

bash

clang -g -O2 -target bpf -D__TARGET_ARCH_x86 \
  -c count_exec.bpf.c -o count_exec.bpf.o
bpftool gen skeleton count_exec.bpf.o > count_exec.skel.h

go deeper

for a junior

Know that eBPF programs are written in restricted C and must get into the kernel somehow, and be able to say that bcc compiles that C on the machine while libbpf compiles it ahead of time.

for a middle

Explain the mechanics both ways: bcc's embedded Clang plus kernel headers versus a -g compile that emits BTF and relocation records which libbpf patches against the running kernel at load.

for a senior

Argue the operational case — no compiler or headers on production hosts, millisecond startup, one binary to distribute — and name the one thing CO-RE requires in exchange, kernel BTF, plus what you do on hosts that lack it.

for a principal

Own the standard: which model your organisation ships, the minimum kernel and BTF baseline that implies, and how you keep an escape hatch for hosts that fall outside it without maintaining two codebases forever.

The problem both approaches are solving

A BPF program that reads kernel data structures is compiled against a particular kernel's headers. Field offsets inside structures like task_struct move between kernel versions and even between distro configurations, because a CONFIG_ option that adds one member shifts everything after it. A binary compiled on one kernel and run on another would silently read the wrong bytes. bcc and libbpf answer this in two completely different ways.

bcc: compile on the target, every time

In bcc the BPF C program is data — a string in the Python source, or a file read at startup. The BPF() constructor hands that string to an LLVM/Clang instance linked into libbcc, which compiles it against the headers of the currently running kernel and then loads the result via the bpf() syscall.

python

from bcc import BPF
b = BPF(text="int hello(void *ctx) { return 0; }")

Because compilation happens on the box, offsets are correct by construction. That is genuinely elegant for exploratory work, and it is why bcc dominated the early tracing-tool era. The costs land on operations:

Toolchain on production hosts. libbcc drags in LLVM and Clang — a large dependency for a machine that is supposed to run one application.
Kernel headers must be present and must match. They come from the distro's kernel-headers/kernel-devel package under /lib/modules/$(uname -r)/build, or from the kheaders module, which exposes them as /sys/kernel/kheaders.tar.xz. On a minimal or immutable image they are usually absent, and after a kernel upgrade with no reboot they are the wrong ones.
Startup latency and memory. Compiling takes seconds and the compiler's working set is large, which is unpleasant when you are trying to catch a short-lived incident and awful when the tool runs on thousands of hosts.
Failure mode is a build error at 3am, in a language most on-call engineers will not debug under pressure.

libbpf and CO-RE: compile once, relocate at load

CO-RE means "Compile Once — Run Everywhere". You build the BPF object ahead of time:

bash

clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -c probe.bpf.c -o probe.bpf.o

-g is what matters: it makes Clang emit BTF, a compact type-description format, into the object. When the source accesses a kernel struct field through libbpf's helpers — BPF_CORE_READ(), or the __builtin_preserve_access_index machinery behind it — the compiler also records a relocation: "this instruction loads field pid of struct task_struct; patch its offset at load time".

At run time libbpf reads the running kernel's own BTF from /sys/kernel/btf/vmlinux, matches each recorded type and field against it, and rewrites the offsets in the instruction stream before handing the program to the verifier. The same binary therefore loads on kernels whose layouts differ, without a compiler ever touching the host. You typically ship a single statically linked executable with the BPF object embedded in it via a generated skeleton header.

The trade you are actually making

	bcc	libbpf + CO-RE
Compiler on target	required	not required
Kernel headers on target	required	not required
Kernel BTF on target	not required	required
Startup	seconds	milliseconds
Distribution unit	source + runtime + LLVM	one binary

The single thing CO-RE demands that bcc does not is kernel BTF, which exists only when the kernel was built with CONFIG_DEBUG_INFO_BTF=y. Mainstream distro kernels have shipped it since roughly 2020, but very old or custom-built kernels may not, and then a CO-RE binary cannot resolve its relocations.

Where each still belongs

Use bcc's Python API for one-off exploration on a lab box where a toolchain is fine and iteration speed is everything. Use libbpf CO-RE for anything you ship: agents, always-on collectors, tools handed to other teams. The bcc project itself acknowledges this — its libbpf-tools directory contains CO-RE rewrites of the classic tools precisely so they can be distributed as plain binaries.

## The problem both approaches are solving A BPF program that reads kernel data structures is compiled against a particular kernel's headers. Field offsets inside structures like `task_struct` move between kernel versions and even between distro configurations, because a `CONFIG_` option that adds one member shifts everything after it. A binary compiled on one kernel and run on another would silently read the wrong bytes. bcc and libbpf answer this in two completely different ways. ## bcc: compile on the target, every time In bcc the BPF C program is data — a string in the Python source, or a file read at startup. The `BPF()` constructor hands that string to an LLVM/Clang instance linked into libbcc, which compiles it against the headers of the *currently running* kernel and then loads the result via the `bpf()` syscall. ```python from bcc import BPF b = BPF(text="int hello(void *ctx) { return 0; }") ``` Because compilation happens on the box, offsets are correct by construction. That is genuinely elegant for exploratory work, and it is why bcc dominated the early tracing-tool era. The costs land on operations: - **Toolchain on production hosts.** libbcc drags in LLVM and Clang — a large dependency for a machine that is supposed to run one application. - **Kernel headers must be present and must match.** They come from the distro's kernel-headers/`kernel-devel` package under `/lib/modules/$(uname -r)/build`, or from the `kheaders` module, which exposes them as `/sys/kernel/kheaders.tar.xz`. On a minimal or immutable image they are usually absent, and after a kernel upgrade with no reboot they are the *wrong* ones. - **Startup latency and memory.** Compiling takes seconds and the compiler's working set is large, which is unpleasant when you are trying to catch a short-lived incident and awful when the tool runs on thousands of hosts. - **Failure mode is a build error at 3am**, in a language most on-call engineers will not debug under pressure. ## libbpf and CO-RE: compile once, relocate at load CO-RE means "Compile Once — Run Everywhere". You build the BPF object ahead of time: ```bash clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -c probe.bpf.c -o probe.bpf.o ``` `-g` is what matters: it makes Clang emit BTF, a compact type-description format, into the object. When the source accesses a kernel struct field through libbpf's helpers — `BPF_CORE_READ()`, or the `__builtin_preserve_access_index` machinery behind it — the compiler also records a *relocation*: "this instruction loads field `pid` of `struct task_struct`; patch its offset at load time". At run time libbpf reads the running kernel's own BTF from `/sys/kernel/btf/vmlinux`, matches each recorded type and field against it, and rewrites the offsets in the instruction stream before handing the program to the verifier. The same binary therefore loads on kernels whose layouts differ, without a compiler ever touching the host. You typically ship a single statically linked executable with the BPF object embedded in it via a generated skeleton header. ## The trade you are actually making | | bcc | libbpf + CO-RE | |---|---|---| | Compiler on target | required | not required | | Kernel headers on target | required | not required | | Kernel BTF on target | not required | required | | Startup | seconds | milliseconds | | Distribution unit | source + runtime + LLVM | one binary | The single thing CO-RE demands that bcc does not is kernel BTF, which exists only when the kernel was built with `CONFIG_DEBUG_INFO_BTF=y`. Mainstream distro kernels have shipped it since roughly 2020, but very old or custom-built kernels may not, and then a CO-RE binary cannot resolve its relocations. ## Where each still belongs Use bcc's Python API for one-off exploration on a lab box where a toolchain is fine and iteration speed is everything. Use libbpf CO-RE for anything you *ship*: agents, always-on collectors, tools handed to other teams. The bcc project itself acknowledges this — its `libbpf-tools` directory contains CO-RE rewrites of the classic tools precisely so they can be distributed as plain binaries.

What does the -g flag actually add to the compiled BPF object, and why does CO-RE stop working without it?
`-g` makes Clang emit BTF type information plus the CO-RE relocation records into the `.o`. Those records are what libbpf consults at load time to decide which instructions carry a struct field offset that needs patching. Without `-g` the object has no BTF, libbpf has nothing to relocate against the kernel's types, and any access to a kernel structure keeps the build machine's hardcoded offsets.
Does CO-RE protect you when a kernel renames or removes a struct field entirely, not just moves it?
No. Relocation adjusts offsets for fields that still exist; it cannot invent a field that was removed or renamed. libbpf will fail the relocation and the load. You handle that in source with `bpf_core_field_exists()` and take a different code path, or read an alternative field, so one binary can cope with both layouts.
Why do bcc-based tools sometimes fail on a host that has just been kernel-upgraded but not rebooted?
bcc compiles against the headers it finds for the running kernel. After a package upgrade the headers on disk describe the *new* kernel while the *old* one is still running, so struct layouts can disagree — the tool either fails to build or, worse, reads fields at the wrong offsets. A CO-RE binary is immune because it relocates against the live kernel's BTF.

saying these in an interview costs you the question

Saying bcc ships precompiled BPF bytecode like libbpf does
Claiming CO-RE recompiles the program on each host
Thinking kernel headers are needed for a CO-RE binary
Assuming BTF is present on every Linux kernel
Believing bcc's runtime compile is free because it is cached

Two ways to build an eBPF tool on Linux are the bcc framework and libbpf with CO-RE. Where does each one compile the BPF C program, and what does that difference cost you when you ship the tool to a fleet of servers?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

The problem both approaches are solving

bcc: compile on the target, every time

libbpf and CO-RE: compile once, relocate at load

The trade you are actually making

Where each still belongs

▾common follow-ups3 questions

▾red flags5 flags

§ context

▾related questions7 questions

▾related topics4 topics

The problem both approaches are solving

bcc: compile on the target, every time

libbpf and CO-RE: compile once, relocate at load

The trade you are actually making

Where each still belongs

answer

go deeper

context