skip to content

In C++, how do you stop two per-thread std::atomic counters in one struct from slowing each other down, and what does std::hardware_destructive_interference_size promise?

level: middleimportance: should knowfreq 52%

answer

  1. two counters, one cache line
  2. alignas on each member
  3. a C++17 constant in <new>
  4. a recommendation, not the line size
  5. better still: count locally

basics

~10 s

Give each counter its own cache-line-sized slot with alignas(std::hardware_destructive_interference_size), or count in a local variable and publish once. That C++17 constant in <new> is only the implementation's recommended minimum offset between concurrently written objects.

solid answer

~50 s

Two `std::atomic<long>` members of one struct sit 8 bytes apart, so they share a cache line, and each thread's write takes the line away from the other core: **false sharing**. The C++ fix is layout: `alignas(std::hardware_destructive_interference_size)` on each member puts them in separate slots, and since C++17 libc++ and libstdc++ allocate over-aligned types through aligned `operator new`, so a `std::vector` of such structs stays aligned. The constant comes from `<new>` (C++17) and the standard calls it the **minimum recommended offset** between two concurrently accessed objects, implementation-defined and at least `alignof(std::max_align_t)`. Clang sets it to 64 for x86-64 and 256 for AArch64; GCC derives it from `-mtune` and warns by default with `-Winterference-size` because the value can differ between builds, so it does not belong in a library's public ABI. Often the best fix is to accumulate in a local and write the shared counter once.

code

cpp · 32 lines
cpp
#include <atomic>
#include <chrono>
#include <cstdio>
#include <functional>
#include <new>
#include <thread>

struct Packed {  // both counters share one cache line
  std::atomic<long> a{0};
  std::atomic<long> b{0};
};
struct Apart {   // each counter starts its own interference-sized slot
  alignas(std::hardware_destructive_interference_size) std::atomic<long> a{0};
  alignas(std::hardware_destructive_interference_size) std::atomic<long> b{0};
};

template <class Counters> double seconds() {
  Counters c;
  auto work = [](std::atomic<long>& n) {
    for (int i = 0; i < 50'000'000; ++i) n.fetch_add(1, std::memory_order_relaxed);
  };
  auto t0 = std::chrono::steady_clock::now();
  std::thread t1(work, std::ref(c.a)), t2(work, std::ref(c.b));
  t1.join(); t2.join();
  return std::chrono::duration<double>(std::chrono::steady_clock::now() - t0).count();
}

int main() {
  std::printf("interference size %zu, sizeof(Packed) %zu, sizeof(Apart) %zu\n",
              std::hardware_destructive_interference_size, sizeof(Packed), sizeof(Apart));
  std::printf("packed %.2fs, apart %.2fs\n", seconds<Packed>(), seconds<Apart>());
}

go deeper

for a junior

Recall that two variables written by different threads can slow each other down just by sitting next to each other in memory.

for a middle

Show the alignas fix with std::hardware_destructive_interference_size, say it is C++17 and in <new>, and explain why the struct grows.

for a senior

Prefer removing the shared writes (local accumulation), confirm with timing and profiler events, and pin the layout with static_assert.

for a principal

Weigh the ABI risk: an implementation-defined, tuning-dependent value has no place in a public type, so choose and document a fixed padding there.

## The symptom in C++ ```cpp struct Stats { std::atomic<long> renderedFrames{0}; // written by the render thread std::atomic<long> simulatedTicks{0}; // written by the simulation thread }; ``` The two members are adjacent, 8 bytes apart, so they almost always share one cache line. Each `fetch_add` needs that line in exclusive state on the writing core, so the line bounces between the two cores on every increment. Neither thread reads the other's counter, there is no data race and no lock, yet two threads run slower than one would. That is **false sharing**: sharing a line, not a variable. How coherence protocols move lines between cores is covered with cache coherence; what follows is how C++ lets you control the layout. ## The C++17 tools C++17 added two constants to `<new>`: | Name | What the standard says | Typical use | |---|---|---| | `std::hardware_destructive_interference_size` | the **minimum recommended offset** between two concurrently accessed objects to avoid contention | separate data written by different threads | | `std::hardware_constructive_interference_size` | the **maximum recommended size** of memory holding objects used together by concurrent threads | keep data read together inside one line | Both are `inline constexpr std::size_t`, **implementation-defined**, and at least `alignof(std::max_align_t)`; the feature-test macro is `__cpp_lib_hardware_interference_size`. They are recommendations, not a promise that a cache line has that size. Values in practice: - **Clang** sets destructive and constructive to 64 for x86-64. For AArch64 it sets destructive to 256 and constructive to 64. - **GCC** sets them from the current `-mtune` or `-mcpu`, typically the target CPU's L1 line size, and documents that every translation unit that relies on them for ABI compatibility must use the same tuning. ## Applying it 1. **Align each hot member.** `alignas(std::hardware_destructive_interference_size)` on each counter starts each one on its own slot boundary; the struct's size grows to a multiple of that alignment (512 bytes for two counters on AArch64 with Clang). 2. **Align per-thread array elements.** For `std::vector<PerThread>` put the `alignas` on the element type. Since C++17, `new` passes an over-aligned type's alignment to `operator new`, and both libc++ and libstdc++ make `std::allocator` do the same, so every element really starts on the boundary. 3. **Avoid writing shared memory at all.** Accumulate into a local variable, or a `thread_local`, and add the total to the shared atomic once per batch or frame. This removes the traffic instead of spreading it out. 4. **Keep read-mostly data apart from write-hot data.** A configuration value read by every thread should not share a line with a counter one thread writes. ## Per-thread slots in an array The same problem appears when each worker thread owns one element of an array of statistics: ```cpp struct alignas(std::hardware_destructive_interference_size) WorkerStats { long jobsDone = 0; long bytesProcessed = 0; }; std::vector<WorkerStats> stats(workerCount); // worker i writes only stats[i] ``` Without the `alignas`, `stats[0]` and `stats[1]` are 16 bytes apart and share a line. With it, `sizeof(WorkerStats)` rounds up to the alignment, so consecutive elements start on separate slots. Two fields written by the **same** thread can stay together inside one slot; padding only has to separate data written by **different** threads. ## The ABI trap Because the value depends on compiler, target and tuning flags, using it in a type that crosses a library boundary can give two binaries different layouts for the same struct. GCC therefore warns about uses of these constants by default under `-Winterference-size`, and its manual advises that a library header for which ABI stability matters should probably not use them at all. Inside one application built with one set of flags, they are the portable spelling. For a public type, a fixed `alignas(64)` or `alignas(128)` chosen deliberately and documented as part of the ABI is the safer choice. ## Confirming the diagnosis - Time the code with the counters packed and padded; on the author's AArch64 machine the padded version of the example took about a quarter of the time. - Use the profiler's cache-to-cache transfer or contention events where the platform offers them, and look at which addresses collide. - Check `sizeof` and `alignof` of the struct with a `static_assert`, so a later edit that drops the `alignas` breaks the build instead of the frame time.

  • Why does a std::vector of padded per-thread counters keep each element aligned in C++17 but not reliably in C++14?
    C++17 added `operator new(std::size_t, std::align_val_t)`, and a new-expression for a type whose alignment exceeds `__STDCPP_DEFAULT_NEW_ALIGNMENT__` passes that alignment to it; libc++ and libstdc++ route `std::allocator` through the same overload. Before C++17, plain `operator new` only promised fundamental alignment, so an `alignas(64)` element type could land on a 16-byte boundary.
  • When is hardware_constructive_interference_size the constant you want instead?
    When several small objects are always used together by concurrent threads, for example an atomic flag and the value it guards: keeping them within that size, and aligning the group to its own size, lets one cache-line transfer bring all of them. The standard's example asserts `sizeof(together) <= hardware_constructive_interference_size`.

  • std::hardware_destructive_interference_size is guaranteed to equal the CPU's cache-line size.
  • Making the counters std::atomic already stops them from interfering with each other.
  • Switching the increments to memory_order_relaxed removes false sharing.
  • alignas on the struct alone keeps its two members on separate cache lines.
  • Padding is always the best fix, even when a thread could count in a local variable.