You own the eBPF-based tracing tools for a fleet of Linux servers spanning several kernel versions and distributions. How would you decide between shipping bcc-based tools and libbpf CO-RE binaries, and what would you standardise in the build and distribution pipeline?

## Frame the decision as a fleet dependency, not a preference The question is not which toolchain is nicer to write. It is what each one obliges you to install and maintain on every production host, forever. bcc puts a compiler in production. That means LLVM and Clang packages, kernel headers that must track every kernel upgrade, seconds of startup latency, and a large transient memory footprint on machines that are already sized for their application. It also means the failure mode during an incident is a compile error. As a fleet-wide standard that is a bad trade. libbpf CO-RE moves all of that to build time. The artifact is a single binary; the host needs only a kernel that exposes BTF. Startup is milliseconds, the memory cost is the program and its maps, and distribution is the same signed-package pipeline you already run for everything else. So the default is clear. The interesting work is in the constraints that choice creates. ## Declare a baseline and inventory the exceptions Write down the minimum kernel version and the requirement that `/sys/kernel/btf/vmlinux` exists. Then measure it, rather than assuming — a one-line check across the fleet tells you what fraction is actually eligible: ```bash test -e /sys/kernel/btf/vmlinux && echo ok || echo no-btf ``` For hosts that fail, there are three honest positions and you must pick one per host class: upgrade the kernel; supply an externally generated BTF for that exact build and load it via `btf_custom_path`; or declare the host out of scope for shipped tooling and handle it by hand. What you must not do is leave it undecided, because the discovery then happens during an incident. ## Standardise the build - **Pin the toolchain.** Clang version, libbpf version and bpftool version live in a build container image, not on developers' machines. eBPF codegen is version-sensitive and "works on my laptop" is a real failure here. - **Generate, do not commit.** `vmlinux.h` and the skeleton headers are build artifacts, regenerated every build. - **Link statically where you can.** The point of CO-RE is one artifact; a binary that needs a particular libbpf on the host has given some of that back. - **One repository, one release train.** Tools should ship as versioned packages with the same signing and rollout controls as any other production software — including the ability to roll back. ## Test where it actually runs This is the part teams skip. A CO-RE binary compiles once, but its *acceptance* depends on the target kernel: which hooks exist, which helpers are available, and what the kernel's verifier will pass. A build-only CI gate proves nothing. So the pipeline should boot a VM per kernel version present in the fleet and, for each, load and attach the binary and assert it produces output. `bpftool feature probe` on each image gives you a machine-readable capability inventory to compare against. When a hook is missing on an older kernel, handle it in code — disable that program before load with `bpf_program__set_autoload(..., false)` and degrade the tool's output — rather than shipping a binary that fails to load on a third of the estate. ## Operate what you ship - **Attribution.** Agree a pin naming convention under `/sys/fs/bpf` so an engineer who finds an unfamiliar program with `bpftool prog show` can trace it to an owner in seconds. - **Resource budget.** Map `memlock` is real kernel memory and a badly sized `max_entries` is charged whether or not the map is populated. Give tools a budget and check it in review. - **Overhead honesty.** A tracing program runs in a hot kernel path. Measure the cost of the hooks you ship on a representative workload before fleet rollout, not after. - **A kill switch.** Because pinned or hook-attached programs outlive their loader, define how a tool is disabled fleet-wide without a reboot. ## Where bcc keeps its place None of this makes bcc worthless. On a lab box, or on a single host during a deep investigation where a toolchain is already present, the iteration speed of editing a Python file and rerunning is genuinely faster than a compile-and-deploy loop. The rule I would state is: exploration may use anything; anything that lands on a production host is a CO-RE binary from the release pipeline. That keeps the fast path fast without letting a compiler leak into the fleet.