An eBPF program counts events by looking a key up in a BPF_MAP_TYPE_HASH and doing (*val)++, and under load the totals come out too low. What is wrong, and what do __sync_fetch_and_add() and BPF_MAP_TYPE_PERCPU_HASH each change about it?

## What the increment really is ```c __u64 *val = bpf_map_lookup_elem(&counts, &key); if (!val) return 0; (*val)++; ``` The pointer points *into the map's storage*, which is why writing through it updates the stored value with no second helper call. But `(*val)++` is three machine steps: load the value, add one, store it back. The verifier is content — nothing unsafe is happening, no memory is touched out of bounds. It is simply not atomic. ## Why that loses counts BPF programs run concurrently. A tracepoint on `sys_enter_openat` fires on whichever CPU the calling task is on, and on a 64-core box that is sixty-four possible simultaneous executions of your program. If two of them are counting the same key, both load 100, both add one, both store 101. Two events, one increment recorded. The kernel does not lock map values for you; there is no implicit critical section around a value pointer. The symptom is characteristic: totals that are close but consistently low, worse the busier the box and the hotter the key, and impossible to reproduce on an idle test machine. If a counter cross-checks against another source and is quietly 3% short under load, this is the first thing to suspect. ## Fix one: make the update atomic ```c __sync_fetch_and_add(val, 1); ``` Clang compiles this GCC-style builtin into a BPF atomic add instruction, and the kernel executes it as an atomic operation on the map's storage. The count becomes correct. What you pay is cache-line contention: every CPU updating the same key is bouncing that line between cores, which on a very hot key can cost more than the work you are instrumenting. It is still the right answer when you need a single shared value — a lock-free shared histogram bucket, a limit that a network program must enforce consistently rather than approximately. ## Fix two: give every CPU its own copy `BPF_MAP_TYPE_PERCPU_HASH` and `BPF_MAP_TYPE_PERCPU_ARRAY` allocate `value_size` bytes **per possible CPU** for each key. When the BPF program looks a key up, the helper returns the pointer to *this CPU's* copy, so: ```c (*val)++; /* safe: nobody else can touch this CPU's copy */ ``` is correct with no atomic and no contention at all. This is why per-CPU maps are the default for high-frequency counters and histograms — bcc and bpftrace lean on them heavily. The cost is threefold: **Memory.** A 64-bit counter on a 96-CPU machine is 768 bytes per key, before you consider that per-CPU values are padded to 8-byte alignment. A large-keyspace per-CPU map is far more expensive than its `value_size` suggests. **The reader has to aggregate.** A user-space lookup on a per-CPU map does not return one value; it fills a buffer holding one value per possible CPU, and your code sums them. libbpf offers `libbpf_num_possible_cpus()` for the sizing — and note *possible*, not *online*: the kernel allocates for CPUs that could come online, and undersizing that buffer corrupts memory in your own agent. **No consistent snapshot.** Summing walks the per-CPU values one at a time while the program keeps incrementing, so the total is a smear across a short interval rather than a point-in-time reading. For rates and histograms this is fine. For an exact invariant it is not. ## Compound updates Both fixes cover a single arithmetic update. If you must read a value, decide something, and write back — a rate limiter checking a budget, say — neither an atomic add nor a per-CPU copy expresses that. Some map types support `bpf_spin_lock()` and `bpf_spin_unlock()` around a lock embedded in the value struct, but the mechanism is narrow: it is unavailable to tracing program types, values holding a lock have restrictions on how user space may read them, and only one lock may be held at a time. Where you cannot use it, redesign so that each CPU decides locally, or move the decision to user space. ## Choosing High-frequency counting and latency histograms: per-CPU, plain increment, sum in the reader. A value that several CPUs must agree on exactly, or a keyspace so large that multiplying by the CPU count is unaffordable: shared map with `__sync_fetch_and_add()`. What is never right is a plain `++` on a shared map — it is not a performance optimisation, it is a data loss bug that only shows up under the load you built the tool to measure.