skip to content

A third-party Linux driver that you compiled and installed by hand stops loading after the server is upgraded to a newer kernel. Why does that happen, what does DKMS change about it, and what extra requirement appears on a UEFI Secure Boot machine?

level: seniorimportance: nice to knowfreq 35%

answer

  1. one module tree per kernel release
  2. no stable in-kernel ABI
  3. store the source, not the binary
  4. rebuild needs matching headers
  5. a second gate on verified-boot machines

basics

~20 s

Modules are installed per kernel release under /lib/modules and carry a version-magic string checked at load, so a module built for the old kernel is neither present nor acceptable for the new one. DKMS keeps the source and rebuilds it automatically for each installed kernel.

solid answer

~60 s

Two things break at once. The new kernel has its own `/lib/modules/<release>` tree, and your hand-built `.ko` is not in it — so the module is simply not found. Even if you copied it across, every module carries a *vermagic* string recording the kernel release and key build options, and the kernel rejects a mismatch with `Invalid module format`, logging `version magic … should be …`. Kernel internals are not a stable ABI, so this check is protecting you from memory corruption, not being pedantic. DKMS solves it by keeping the driver *source* registered on the machine with a `dkms.conf`, and hooking kernel package installation: whenever a new kernel is installed it rebuilds the module against that kernel's headers, installs it into the new tree and reruns `depmod`. That requires the matching headers package to be present. On a Secure Boot machine there is a second gate: the kernel enforces module signatures, and an unsigned module fails with `Key was rejected by service`, so the freshly built module must be signed with a key enrolled as a Machine Owner Key. Either way, an out-of-tree module taints the kernel.

go deeper

for a junior

Know that modules are installed per kernel version and that a driver built for one kernel does not work on another. Recognise DKMS as the mechanism that rebuilds third-party drivers when the kernel changes.

for a middle

Explain vermagic and why an unstable in-kernel ABI makes the check necessary, describe how DKMS registers source plus a build recipe and hooks kernel installation, and name the headers package as the prerequisite.

for a senior

Diagnose from the error text — absent module, invalid format, rejected key — and know the Secure Boot path end to end: generate a key, enrol it as a MOK with a human at the firmware prompt, sign each rebuild. Say plainly what taint means for support.

for a principal

Own the policy: out-of-tree drivers couple your kernel upgrade cadence to a vendor's build, taint the kernel and complicate verified boot. Decide deliberately whether to accept that, keep key material for signing under management, or refuse hardware that needs it.

## Why the module vanishes Modules are keyed by kernel release. `/lib/modules/6.1.0-x/` and `/lib/modules/6.5.0-y/` are separate trees with separate `modules.dep` indexes, and `modprobe` only ever looks in the tree for the *running* kernel. Installing a new kernel therefore never carries a hand-installed module forward — nothing copies it, and nothing warns you. The first symptom is a driver that is silently absent after a reboot. Copying the file across does not rescue you either, because of version magic. ## Version magic and the absent ABI Every module embeds a `vermagic` string produced at build time: the kernel release plus a handful of configuration attributes such as SMP and preemption settings. At load time the kernel compares it with its own. On a mismatch the insert is refused — the tool reports `Invalid module format` and the kernel log carries the specific `version magic '…' should be '…'` line. The check exists because Linux deliberately has no stable in-kernel ABI. Structure layouts, inline functions and locking rules change between releases, so a module compiled against different headers can be reading a field at the wrong offset. The vermagic mismatch converts a silent memory-corruption bug into a clean refusal at load time. Distribution kernels usually add a stricter per-symbol version check on top of it, which is why even a matching release string is not always enough across a rebuilt kernel. ## What DKMS actually does DKMS — Dynamic Kernel Module Support — inverts the model: instead of storing a *built* module, the machine stores the driver's *source*, conventionally under `/usr/src/<name>-<version>/`, alongside a `dkms.conf` that names the module, its version and how to build it. Registering it with the DKMS tooling puts it in a database, and DKMS installs hooks that run when a kernel package is installed or removed. On the next kernel install, the hook rebuilds the module against that kernel's headers, installs the result into the new kernel's module tree, and reruns `depmod` so `modprobe` can find it. `dkms status` shows which kernels currently have a built copy. The standing prerequisite is the headers package for each kernel — `linux-headers-<release>` on Debian and Ubuntu, `kernel-devel` on RHEL, Fedora and SUSE — plus a compiler and make. The classic failure is a host whose headers package was never installed or has drifted from the running kernel: the build fails during the kernel upgrade, often in a package-manager transaction nobody is reading, and the driver is gone on reboot. ```bash dkms status # which modules are built for which kernels modinfo -F vermagic <module> # what kernel this module was built for ``` ## Taint An out-of-tree module sets a taint flag in the kernel, visible as a bitmask, and a module with a non-GPL-compatible licence declaration sets a different one. Taint does not change the kernel's behaviour; it is recorded and printed alongside any oops so that whoever reads the crash knows unsupported code was resident. Practically it is a support boundary: distribution vendors will ask you to reproduce without the tainting module. Being able to say that plainly — that taint is a support and forensics signal, not a runtime penalty — is a real differentiator in an interview. ## The Secure Boot layer On a machine booting with UEFI Secure Boot, the distribution's kernel enforces module signatures. Modules shipped by the distribution are signed with a key built into that kernel. A module you built is not, and the load fails with `Key was rejected by service` — a distinct failure from the vermagic case, and one that people misdiagnose as a build problem for hours. The supported way through is a Machine Owner Key: generate a key pair, enrol the public half with `mokutil --import`, confirm the enrolment in the firmware prompt on the next boot, and sign each built module with the private half using the kernel's `sign-file` helper. Debian and Ubuntu integrate this with DKMS, which can sign each rebuild automatically once a MOK is enrolled. The alternative — disabling Secure Boot — trades the whole verified-boot chain for the convenience of one driver, which is a decision worth naming as a decision. ## The complete answer The module is per-kernel and version-checked, so hand-installed drivers do not survive kernel upgrades. DKMS moves the artefact from a built binary to registered source plus a rebuild hook, which is why every serious third-party driver ships that way. Secure Boot adds signature enforcement on top, satisfied by enrolling your own key rather than by rebuilding harder. And every out-of-tree driver taints the kernel, which is what the vendor will point at when you file a bug.

  • How do you tell a signature rejection apart from a version-magic mismatch when a module refuses to load?
    By the error. A vermagic mismatch reports an invalid module format, and the kernel log carries an explicit "version magic … should be …" line naming both strings. A signature rejection under Secure Boot reports that the key was rejected by service. The first is fixed by rebuilding against the right headers, the second only by signing with an enrolled key — treating them the same wastes hours.
  • Does DKMS help at all on a Secure Boot machine with no enrolled key?
    It still rebuilds the module for each new kernel, so the vermagic problem is solved, but the result is unsigned and the kernel refuses to load it. On Debian and Ubuntu the DKMS packaging can generate and sign with a MOK, but the enrolment step needs a human at the firmware prompt on first boot. Until that is done the driver is built and unusable.
  • Why does the kernel care about vermagic when the release number already matches?
    Because the release string is not the whole build. Vermagic also records options such as SMP and preemption that change structure layouts and locking, and distribution kernels layer per-symbol version checksums on top. Two kernels can share a release string and still be incompatible, so the check is over the build, not the version number.
  • What is the practical consequence of a tainted kernel when you open a support case?
    The vendor sees the taint flags in the oops output and knows unsupported code was loaded — out-of-tree, proprietary, unsigned, or a forced operation. They will normally ask you to reproduce without it before investigating. It does not degrade the running system; it scopes who owns the bug, which is exactly why the flag is recorded and sticky for the boot.

saying these in an interview costs you the question

  • Copies a .ko between kernel versions and expects it to load
  • Thinks Linux has a stable in-kernel driver ABI
  • Believes DKMS stores a prebuilt binary per kernel
  • Confuses a signature rejection with a build failure
  • Disables Secure Boot as the first response to a rejected module

context

resolving links…