skip to content

Performance and Latency

Low-latency and game C++ is judged on cache misses, allocations and branches more than on big-O. Interviewers ask how you would make a hot path faster and prove it.

on this pageshow

explore

questions

18

In C++, does declaring a function inline make the compiler inline its calls, and what actually decides whether a call is inlined?

level: juniorimportance: must knowfreq 62%

answer

  1. a preference the compiler may ignore
  2. one definition in many translation units
  3. the body must be visible at the call
  4. a cost model with a threshold
  5. ask the compiler for remarks

basics

~20 s

No. The inline specifier's binding effect is that a function may be defined in several translation units; the standard calls inline substitution only a preference. The optimiser inlines a call when it can see the body and its cost model approves.

solid answer

~50 s

`inline` does two things. Its **guaranteed** effect is on the one-definition rule: an inline function may be defined, identically, in every translation unit that includes its header, and is still one entity with one address. Its effect on code generation is only a hint: the standard says inline substitution "is to be preferred" and that an implementation "is not required to perform" it. What decides inlining is the optimiser: whether the **definition is visible** in this translation unit (a function defined in another `.cpp` cannot be inlined without link-time optimisation), whether the **target is known** (a virtual call or a function pointer only after the compiler proves which function it is), and a **cost model** comparing the callee's size against a threshold. Functions without the keyword are inlined all the time, and `[[gnu::always_inline]]` or `[[gnu::noinline]]` override the heuristic. Clang's `-Rpass=inline` and `-Rpass-missed=inline` report each decision.

code

cpp · 18 lines
cpp
#include <cstdio>

struct Vec3 { float x, y, z; };

// inline: may be defined in every translation unit that includes it.
// For speed it is only a hint the optimizer is free to ignore.
inline float dot(Vec3 a, Vec3 b) { return a.x * b.x + a.y * b.y + a.z * b.z; }

[[gnu::noinline]] float dotNoInline(Vec3 a, Vec3 b) { return dot(a, b); }

float lengthSquared(Vec3 v) { return dot(v, v); }  // no inline keyword

float apply(float (*f)(Vec3, Vec3), Vec3 a, Vec3 b) { return f(a, b); }

int main(int argc, char**) {
  Vec3 v{float(argc), 2, 3};
  std::printf("%g %g %g\n", lengthSquared(v), dotNoInline(v, v), apply(dot, v, v));
}

go deeper

for a junior

Say that inline is about allowing a definition in a header and that the compiler decides inlining on its own.

for a middle

List what the optimiser needs: a visible definition, a known target, a size under its threshold, and no noinline.

for a senior

Verify instead of assuming: optimisation remarks, the assembly, and a measurement before reaching for always_inline.

for a principal

Set the team rule: write small functions freely, keep hot code's definitions visible, and treat forced inlining as a measured exception.

## What the inline specifier means in C++ The `inline` specifier on a function is mostly about **linkage and the one-definition rule (ODR)**, not speed: - An inline function with external linkage **may be defined in more than one translation unit**, as long as every definition is identical; it is still one entity with one address, and a `static` local variable inside it is one variable. - That is why a function defined in a header must be `inline` (or a template, or `constexpr`) to avoid duplicate-symbol link errors. - Some functions are **implicitly inline**: `constexpr` functions, functions defined inside their class definition in ordinary non-module code, and deleted functions. Since C++20, a member function defined inside a class in a **named module** is no longer implicitly inline. The standard does mention speed, but as a preference: the specifier "indicates to the implementation that inline substitution of the function body at the point of call is to be preferred", and "an implementation is not required to perform this inline substitution". Compilers treat it as a mild hint at most. ## What actually decides whether a call is inlined **Inlining** means replacing a call with a copy of the callee's body. The optimiser decides per call site, and it needs all of the following: 1. **The definition is visible.** If the function is only declared in this translation unit and defined in another `.cpp`, there is no body to copy. Link-time optimisation can lift that limit; its flags belong to the compiler's own documentation. 2. **The target is known.** A direct call is easy. A call through a function pointer, a `std::function` or a virtual function can be inlined only after the optimiser proves which function it is, for example after inlining a caller that passed a known function, as the remarks below show for `apply(dot, ...)`. 3. **The cost model agrees.** Each compiler estimates the callee's size after simplification and compares it with a threshold that depends on the optimisation level and on how much has already been inlined into the caller. Small functions almost always pass; big ones usually do not, whatever their keyword. 4. **Nothing forbids it.** Direct recursion cannot be fully inlined, and a `noinline` attribute blocks it outright. Without optimisation, GCC inlines nothing except functions marked `always_inline`, which is one reason a `-O0` build of abstraction-heavy C++ is so much slower than an optimised one. ## Forcing and forbidding | Spelling | Where | Effect | |---|---|---| | `inline` | standard | ODR permission; a weak hint at most | | `[[gnu::always_inline]]` | GCC and Clang attribute | inline wherever possible, even without optimisation | | `[[gnu::noinline]]` | GCC and Clang attribute | never inline this function | | `__forceinline` | MSVC keyword | strong request to inline | These are tools for measured cases: forcing a large function inline into many call sites can grow code until instruction-cache misses cost more than the calls saved. ## Seeing the decision instead of guessing - **Optimisation remarks.** Clang's `-Rpass=inline` reports each inlined call with its cost and threshold, and `-Rpass-missed=inline` reports the refusals with a reason. - **GCC's `-Winline`** warns when a function declared `inline` could not be inlined. - **Read the assembly**: compile with `-O2 -S`, or paste the code into a compiler explorer, and look for a `call` or `bl` to the function you expected to vanish. ## Why it matters for abstraction cost Inlining is the step that makes small C++ abstractions free: once `dot(a, b)` is inlined, the optimiser sees through the function boundary and can keep values in registers, fold constants and vectorise the surrounding loop. What inlining unlocks, concretely: - **Constant propagation**: arguments that are constants at the call site become constants inside the copied body, and branches on them disappear. - **Dead-code removal**: work whose result the caller ignores is deleted. - **Register allocation across the boundary**: no argument marshalling, no spills around the call. - **Vectorisation**: a loop whose body was a call becomes a loop of arithmetic the compiler can widen. A call that is not inlined is an optimisation barrier for all of that, which is why the useful question in a performance review is "was this call inlined?", not "is this function marked `inline`?"

  • Why does a function defined in a header without inline cause a link error, and why does a template not?
    Each translation unit that includes the header gets its own definition with external linkage, and the linker sees several definitions of one ordinary function, which the one-definition rule forbids. Declaring it `inline` makes multiple identical definitions legal. Function templates are exempt in the same way: their instantiations may appear in many translation units and the toolchain merges them.
  • When could forcing a function inline make a hot loop slower?
    When the function is large or called from many places, copying its body everywhere grows the code; the hot loop may no longer fit the instruction cache or the decoded-instruction cache, and register pressure inside the loop rises. Forcing inline is worth doing only where a measurement shows the call itself, or a missed optimisation across it, is the cost.

  • The inline keyword forces the compiler to paste the body at every call.
  • A function without inline is never inlined.
  • inline gives a function internal linkage, so each translation unit gets a private copy.
  • Calls through a function pointer can never be inlined.
  • Marking every small function inline is how you make C++ code fast.

Assumes C++23 wording; the rule that an in-class member definition is implicitly inline holds outside C++20 named modules, where it no longer applies.

open as a page

In a C++ tick handler, why does reusing a reserved std::vector with clear() avoid the heap allocations a fresh vector per tick makes?

level: juniorimportance: must knowfreq 68%

basics

~20 s

A vector declared inside the tick starts with no buffer, reallocates as it grows and frees its buffer at scope exit, every tick. A vector reserved once and cleared each tick keeps its buffer, so push_back within capacity never allocates.

open as a page

In C++, why does updating a std::vector<Particle> usually run much faster than updating a std::vector<std::unique_ptr<Particle>>?

level: juniorimportance: must knowfreq 66%

basics

~20 s

std::vector<Particle> stores the particles back to back in one block, so the loop streams through memory the prefetcher can predict. A vector of std::unique_ptr holds only pointers to separately allocated particles: an extra dependent load, and often a cache miss, per element.

open as a page

Which ordinary C++ operations in a market-data tick path allocate on the heap with no visible new, and how do you catch them?

level: middleimportance: must knowfreq 58%

basics

~20 s

Long std::string values, std::function holding a large callable, vector growth, make_shared, new-key inserts into std::map or std::unordered_map, and string formatting all allocate. Catch them by counting global operator new in a test or with a heap profiler.

open as a page

Why can a Google Benchmark of a C++ JSON parser report almost zero time per iteration, and how does benchmark::DoNotOptimize fix it?

level: middleimportance: must knowfreq 52%

basics

~20 s

Under the as-if rule the optimizer may delete a call whose result is never used, so the timed loop measures nothing. benchmark::DoNotOptimize(result) forces the result into a register or memory, so the parse must actually run each iteration.

open as a page

In C++, which loop order is fast over a float grid[H][W] or its flattened std::vector, and why?

level: juniorimportance: should knowfreq 52%

basics

~20 s

Row by row: outer loop over r, inner loop over c. A C++ float grid[H][W] is an array of H contiguous rows, so grid[r][c + 1] is the next float in memory, while grid[r + 1][c] is W floats away.

open as a page

In C++, why can an if/else clamp over random floats run far slower than a loop that always stores std::clamp's result?

level: middleimportance: should knowfreq 48%

basics

~20 s

The if/else version writes only on out-of-range paths, so the compiler keeps a data-dependent branch per element, which random data mispredicts constantly. Always storing std::clamp's result lets it use min, max or select instructions and vectorise, leaving no branch to mispredict.

open as a page

In C++, why is passing a small trivially copyable Vec3 by value often faster than by const reference, and when is it not?

level: middleimportance: should knowfreq 50%

basics

~20 s

Passed by value, a 12-byte trivially copyable Vec3 usually travels in registers as a private copy the callee never has to reload. By const reference it is an address that might alias other writes. Large or costly-to-copy types still go by reference.

open as a page

How do you turn a C++ particle array-of-structs into a structure-of-arrays, and what does the update loop gain and lose?

level: middleimportance: should knowfreq 50%

basics

~20 s

Replace std::vector<Particle> with a struct holding one std::vector per field, all the same length. A loop that updates positions then streams only x, vx and friends, densely and vectorisably; you lose the Particle object, single-call insertion and cheap whole-particle access.

open as a page

In C++, how do you stop two per-thread std::atomic counters in one struct from slowing each other down, and what does std::hardware_destructive_interference_size promise?

level: middleimportance: should knowfreq 52%

basics

~10 s

Give each counter its own cache-line-sized slot with alignas(std::hardware_destructive_interference_size), or count in a local variable and publish once. That C++17 constant in <new> is only the implementation's recommended minimum offset between concurrently written objects.

open as a page

In C++20, what do the [[likely]] and [[unlikely]] attributes actually change in generated code, and how do teams misuse them?

level: seniorimportance: should knowfreq 35%

basics

~20 s

C++20's [[likely]] and [[unlikely]] mark a statement or label as the hot or cold path, so the compiler makes the hot path the fall-through and moves the cold one out of line. They do not program the CPU's branch predictor.

open as a page

In a C++ 3D math library, summing virtual Shape::area() over std::vector<std::unique_ptr<Shape>> is much slower than a template loop over std::vector<Disc>; what does the template change, and what does it cost?

level: seniorimportance: should knowfreq 45%

basics

~20 s

With the template the element type is known at compile time, so Disc::area is a direct call the compiler inlines and then vectorises. The virtual loop chases a pointer per element and makes an indirect call the optimiser cannot see through.

open as a page

How does C++17's std::pmr::monotonic_buffer_resource over a fixed buffer make a tick's temporary containers allocation-free, and where can that setup still reach the heap?

level: seniorimportance: should knowfreq 32%

basics

~20 s

Build std::pmr containers on a monotonic_buffer_resource placed over a preallocated buffer: allocation bumps a pointer, deallocation does nothing, and everything is released at once. Overflow goes upstream, by default to new and delete, and copies and non-pmr elements escape too.

open as a page

A C++ particle update reads 24 bytes of a 64-byte struct that also holds a std::string debug name; how would you hot/cold split it, and what does the split cost?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Move the fields the per-frame loop never reads into a parallel cold array with the same index, leaving 24-byte hot elements. The cost: both arrays must change in step, and code needing both halves takes an extra cache miss.

open as a page

A C++ JSON parser change cuts the Google Benchmark mean by 30%, yet production p99 parse time got worse; how do you measure and explain the tail?

level: seniorimportance: should knowfreq 40%

basics

~20 s

Google Benchmark reports a mean per iteration, which hides rare slow calls. Time each parse individually over a realistic input mix, compute p99 and max, find which inputs are slow, then profile those inputs alone to see why.

open as a page

A pull request claims its C++ JSON parser change is 30% faster on one Google Benchmark run; how do you check the speedup is real?

level: seniorimportance: should knowfreq 45%

basics

~20 s

Build both versions identically, run them on a quiet machine with repetitions and warm-up, compare them with Google Benchmark's compare.py U test, check the inputs are realistic and nothing was optimized away, then explain the gain with a profiler.

open as a page

As lead of a C++ market-data handler, how would you set an allocation budget and allocator policy for the tick path, and how would you enforce them?

level: principalimportance: should knowfreq 22%

basics

~20 s

Set the budget at zero heap allocations per tick after warm-up, give each data lifetime its allocator (startup reserves, pools, a per-tick arena), choose throw or counted fallback on overflow, and enforce it with an allocation-counting test and production counters.

open as a page

As lead of a C++ codebase, how would you design a CI performance gate that blocks regressions like a slower JSON parser without flaky failures?

level: principalimportance: should knowfreq 25%

basics

~20 s

Gate a few hot-path benchmarks on dedicated quiet runners, compare base and change in the same job with repetitions, fail only when a slowdown is both significant and above an A/A-calibrated threshold, and track history for slow drift.

open as a page