A libbpf CO-RE binary that works on a current Ubuntu server fails to load on an older Linux host, and `/sys/kernel/btf/vmlinux` does not exist there. What is missing, and what are your options for that host?

## What the missing file means BTF (BPF Type Format) is a compact description of a kernel's types — every struct, its members and their offsets. The kernel exposes its own BTF as a read-only blob at `/sys/kernel/btf/vmlinux`, and one file per loaded module under `/sys/kernel/btf/<module>`. That blob is produced *at kernel build time*: the build compiles with debug info and then runs `pahole` (from the `dwarves` package) to convert DWARF into BTF, all gated behind `CONFIG_DEBUG_INFO_BTF=y`. If the option was off, or the build machine's pahole was too old, the kernel ships without it and the file is simply absent. Check it directly: ```bash ls -l /sys/kernel/btf/vmlinux grep CONFIG_DEBUG_INFO_BTF /boot/config-$(uname -r) bpftool btf dump file /sys/kernel/btf/vmlinux format c | head ``` ## Why the load fails rather than degrading Your binary carries CO-RE relocation records — "the instruction at this offset reads field `X` of `struct Y`; fix its offset for the running kernel". libbpf resolves each one by looking the type up in the kernel's BTF. With no BTF, there is nothing to look up, and libbpf refuses to load. That is the right behaviour: the alternative would be to keep the build machine's offsets and read arbitrary bytes out of kernel structures, producing plausible nonsense. Failing loudly at load is much better than silently wrong tracing data. ## Option 1 — get a kernel with BTF On a supported distribution this is usually a package away: mainstream kernels have enabled the option since roughly 2020, so a newer stock kernel or a hardware-enablement kernel fixes it outright. If you build your own kernels, turn on `CONFIG_DEBUG_INFO_BTF` and make sure the build host has a pahole new enough for your kernel version — a too-old pahole is the classic reason the option is silently dropped during a build. This is the option to push for, because it fixes every CO-RE tool on that host at once instead of one at a time. ## Option 2 — supply BTF from outside BTF for a kernel can be generated anywhere, as long as you have that exact kernel's debug information. The pattern is: obtain the kernel's debug symbols (a distro debuginfo package), run pahole against them to produce a `.btf` file, put that file on the host, and tell libbpf to use it instead of the kernel's own: ```c LIBBPF_OPTS(bpf_object_open_opts, opts, .btf_custom_path = "/lib/btf/vmlinux-4.19.0.btf"); ``` The skeleton's `__open_opts()` entry point takes the same options struct. Doing this per kernel version for a whole fleet is tedious, which is exactly why the community BTFHub project publishes prebuilt BTF blobs for older distro kernels — the file is generated once and shipped alongside the tool. bpftool can also produce a **reduced** BTF containing only the types a given object actually needs, which shrinks a multi-megabyte blob to something you are willing to embed in a binary. The operational catch: the BTF must match the *exact* kernel build, not merely the version string. A vendor's respin with different config produces different offsets, and using another build's BTF reintroduces the very bug CO-RE prevents. ## Option 3 — do not use CO-RE on that host bcc compiles against the running kernel's headers, so it needs no BTF at all. If the host is a one-off legacy box and you need an answer now rather than a shipping story, installing kernel headers and running the equivalent bcc tool is the pragmatic move. It costs a compiler on the host and slow startup, but it works where CO-RE cannot. ## The judgment an interviewer is listening for The weak answer is "the kernel is too old for eBPF". Usually it is not — the host may run BPF programs perfectly well; it merely cannot describe its own types. Separating those two facts, checking `/boot/config-$(uname -r)`, and knowing that BTF can come from somewhere other than the running kernel is what distinguishes someone who has shipped eBPF tooling across a real fleet from someone who has only run it on their laptop.