skip to content

A numerics kernel built with g++ -O3 -march=native is fast on a dev box but dies with an illegal instruction on a customer's older CPU; why, and how do you fix it?

level: seniorimportance: must knowfreq 48%

answer

  1. instruction set versus scheduling
  2. native means this machine
  3. x86-64-v2, v3, v4 levels
  4. clone hot functions per CPU
  5. FMA changes the bits

basics

~20 s

-march=native lets GCC use every instruction the build machine supports, so the binary executes instructions an older CPU lacks and dies with SIGILL. Build for a baseline -march, tune with -mtune, and clone hot functions with target_clones.

solid answer

~50 s

`-march=cpu` sets the **instruction set** GCC may emit, and the manual warns the code may not run at all on other processors; `-march=native` picks every extension of the **build machine**. The kernel was vectorized with, say, AVX2 and FMA, so the first such instruction on the customer's CPU raises SIGILL. `-mtune=cpu` is different: it only schedules and tunes for a CPU while keeping the baseline instruction set, so it runs everywhere. The fix is to compile for the oldest CPU you support, such as `-march=x86-64-v2` or `-march=x86-64-v3` (x86-64 psABI levels, which use generic tuning), and to recover speed in hot functions with `__attribute__((target_clones("avx2","default")))`, which builds several versions and selects one at load time. Also expect results to change bitwise between targets: GCC 16 defaults C++ to `-ffp-contract=fast`, so a target with FMA fuses multiply-adds and rounds differently.

code

bash · 4 lines
bash
g++ -O3 -march=native -c kernel.cpp            # tied to the build machine's CPU
g++ -O3 -march=x86-64-v2 -mtune=generic -c kernel.cpp   # runs on any x86-64-v2 CPU
echo | g++ -march=x86-64-v3 -dM -E -x c++ - | grep -E '__AVX2__|__FMA__'
g++ -march=native -Q --help=target | grep -E '^ +-march='

go deeper

for a junior

Know that -march=native builds for the build machine's CPU and may not run elsewhere.

for a middle

Explain -march versus -mtune and the x86-64 psABI levels, and choose a baseline for a shipped binary.

for a senior

Diagnose SIGILL after a flag change, recover performance with target_clones, and anticipate FMA-induced result differences in tests.

for a principal

Set the supported-CPU baseline as a product decision, with CI that actually runs on the oldest supported hardware.

## What -march and -mtune control GCC separates two questions about the target CPU: | Option | Controls | Runs on older CPUs? | |---|---|---| | `-march=cpu` | which **instructions** GCC may emit; implies `-mtune=cpu` | **not necessarily** | | `-mtune=cpu` | instruction choice and scheduling **within** the instruction set already allowed | yes | | `-march=native` | every instruction subset the **build machine** supports | only on similar machines | | `-mtune=generic` | tuning for the most common current processors | yes | The manual's wording leaves no doubt: in contrast to `-mtune`, `-march` allows GCC to generate code that **may not run at all** on processors other than the one indicated, and using `-march=native` means the result might not run on different machines. ## The failure At `-O3`, the vectorizer turns the kernel's loops into 256-bit vector code, and with FMA available it fuses `a * x + y` into single instructions. Compiling a simple `y[i] = a * x[i] + y[i]` loop for x86-64 shows the effect: the baseline build uses SSE `mulps` on `%xmm` registers, while the `x86-64-v3` build uses `vfmadd...` instructions on `%ymm` registers. On a CPU without AVX2 or FMA the first such instruction raises an **illegal instruction** fault, SIGILL on Linux, and the process dies, sometimes already at startup if the compiler also used the new instructions in initialisation code. Nothing in the binary checks the CPU first. `-march` is a promise from you to the compiler that the instructions exist. ## Fixes, from simplest to most flexible 1. **Pick a baseline.** Compile for the oldest CPU you support. The x86-64 psABI defines levels GCC accepts directly: - `-march=x86-64`: the original baseline with SSE2; - `-march=x86-64-v2`: adds SSE3 to SSE4.2 and POPCNT; - `-march=x86-64-v3`: adds AVX, AVX2, BMI and FMA; - `-march=x86-64-v4`: adds AVX-512 subsets. These levels have no matching `-mtune`, so GCC uses generic tuning unless you add `-mtune=other-cpu`. 2. **Tune without restricting.** `-mtune=native` or a named CPU changes scheduling for the machines you care about while keeping the baseline instruction set. 3. **Clone the hot functions.** Function multiversioning lets one binary carry several builds of a function: ```cpp __attribute__((target_clones("avx2", "default"))) void axpy(float a, const float* x, float* y, int n) { for (int i = 0; i < n; ++i) y[i] = a * x[i] + y[i]; } ``` GCC compiles one version per option and adds a resolver that picks the best one for the running CPU when the program is loaded. On x86 the options are the same as for the `target` attribute. 4. **Ship several builds** and choose at install or launch time, when whole programs benefit and multiversioning is too fine-grained. ## Choosing inside the code `target_clones` is the declarative form. For coarser decisions GCC offers `__builtin_cpu_supports("avx2")` on x86, which returns whether the running CPU has a feature, so a program can pick an implementation once at startup: - compile the fast kernel in its own file with `-march=x86-64-v3`, and the fallback with the baseline; - call the fast one only when `__builtin_cpu_supports` says the features exist; - keep every other file at the baseline, so nothing outside the kernel can use the new instructions by accident. This keeps the risky instructions in one place you can test on both kinds of machine. ## The numerical side effect Changing `-march` changes more than speed. GCC 16 documents `-ffp-contract=fast` as the default for everything except C in a strict standard mode, so for C++ the compiler may fuse a multiply and an add into one FMA whenever the target has it. An FMA rounds once instead of twice, so a `-march=x86-64-v3` build can produce results that differ in the last bits from a baseline build. Tests that compare floating-point output exactly will flag it. Use tolerances, or `-ffp-contract=off` where bit-for-bit reproducibility across targets matters. ## Checking what you built - `echo | g++ -march=x86-64-v3 -dM -E -x c++ -` shows the feature macros, such as `__AVX2__` and `__FMA__`, that a setting implies. - `g++ -march=native -Q --help=target` reports what `native` resolved to on this machine. - Run the test suite on the oldest supported CPU, or under an emulator that models it, in CI.

  • When is -march=native acceptable?
    When the binary only ever runs on the machine that built it, or on identical hardware: a local experiment, an HPC job compiled on the cluster node type that runs it, or a container image built per fleet. Anything shipped to customers needs a declared baseline instead.
  • Why might the same kernel give slightly different floating-point results at -march=x86-64-v3?
    With FMA available and GCC 16's default `-ffp-contract=fast` for C++, multiply-add pairs can be fused into one instruction that rounds once. The result can differ in the last bits from the baseline build's separate multiply and add, so exact-output tests need tolerances or `-ffp-contract=off`.

  • -mtune=native can make a binary crash on older CPUs.
  • -march=native detects the customer's CPU at run time.
  • GCC inserts a CPU check before using AVX2 instructions under -march.
  • Changing -march never changes floating-point results.
  • An illegal instruction crash after an -march change must be memory corruption.

Assumes GCC 16, which documents -ffp-contract=fast as the C++ default and the x86-64-v2/v3/v4 -march levels.