Profiling a C++ job scheduler shows a heap allocation for each job submitted as a std::function; why does it happen, and how do you remove it?
answer
- type erasure needs storage somewhere
- small buffer is a recommendation
- libc++ and libstdc++ draw the line differently
- copyable target, copies allocate too
- move_only_function, templates, own buffer
basics
~20 sstd::function stores a target of unknown size, so a target too big for its small inline buffer goes on the heap. The standard only recommends avoiding that for small callables. Shrink captures, take a template parameter, or use a fixed-capacity wrapper.
solid answer
~50 sTo store any callable behind one type, `std::function` keeps the target inline only when it fits a small buffer and meets the library's conditions; otherwise it allocates. The standard merely **recommends** avoiding allocation for small callables. On this machine's **libc++** (default ABI), a 16-byte capture stayed inline and a 24-byte one allocated; **libstdc++** (GCC 16) keeps only trivially copyable targets of at most 16 bytes inline, so any `std::string` capture allocates. Copying a `std::function` copies the target and may allocate again. Fixes: capture less (an id or pointer instead of strings), move rather than copy wrappers into the queue, accept callbacks as a **template parameter** where the scheduler can be generic, or use a type-erased job with a fixed inline buffer that rejects oversized callables at compile time. **C++23** `std::move_only_function` accepts move-only targets but still may allocate.
code
cpp · 33 lines#include <array>
#include <cstdio>
#include <cstdlib>
#include <functional>
#include <new>
#include <string>
static int g_allocs = 0; // counts every global operator new call
void* operator new(std::size_t n) {
++g_allocs;
if (void* p = std::malloc(n)) return p;
throw std::bad_alloc{};
}
void operator delete(void* p) noexcept { std::free(p); }
void operator delete(void* p, std::size_t) noexcept { std::free(p); }
template <class F>
void submit(const char* label, F f) {
int before = g_allocs;
std::function<void()> job = std::move(f); // what the scheduler stores
std::printf("%-22s closure=%2zu bytes allocations=%d\n", label, sizeof(F), g_allocs - before);
}
int main() {
int id = 7;
std::array<long, 2> two{};
std::array<long, 3> three{};
std::string tag = "nightly";
submit("one int", [id] { (void)id; });
submit("16 bytes", [two] { (void)two; });
submit("24 bytes", [three] { (void)three; });
submit("std::string", [tag] { (void)tag; });
}go deeper
Know that std::function can allocate memory to store a large callable.
Explain the inline buffer, that its size is implementation-defined, and that copying a std::function copies its target.
Measure allocations per submitted job, explain why libc++ and libstdc++ differ, and choose between shrinking captures, templates and a fixed-capacity job type.
Decide whether the scheduler's public API should erase types at all, and whether a fixed-capacity callable is worth owning as team infrastructure.
## Why erasing the type costs storage A `std::function<void()>` has one fixed size, yet it must hold a lambda capturing four bytes as readily as one capturing a kilobyte. Implementations reserve a small **inline buffer** inside the wrapper and place the target there when it fits; anything else goes into **dynamically allocated** memory owned by the wrapper. The standard's wording is only a recommendation: implementations *should avoid* dynamic allocation for small callable objects, for example a target holding just a pointer to an object and a member function pointer. How small is small is the implementation's choice. ## Where two implementations draw the line | Library | Inline condition (as read from its headers) | Consequence | |---|---|---| | libc++, LLVM 23, default ABI | a buffer of three pointers holds an internal wrapper with its own vtable pointer, and the target must be nothrow copy-constructible | targets up to 16 bytes on a 64-bit target stay inline | | libstdc++, GCC 16 | the target must be trivially copyable, at most `sizeof` a member function pointer (16 bytes on common 64-bit targets) and suitably aligned | any capture of `std::string`, `std::vector` or a `std::shared_ptr` allocates | The code example measured libc++ by replacing the global `operator new`: one `int` and 16 bytes of captures stayed inline, 24 bytes allocated once, and a `std::string` capture allocated once. The string, `"nightly"`, fits libc++'s own small-string buffer, so that single allocation is the wrapper's. libstdc++ was not run here; its rule comes from `__stored_locally` in `bits/std_function.h`. ## Other allocations hiding in the same path 1. **Copies.** Copying a `std::function` copies its target; a heap-stored target is allocated again. Passing wrappers by value through several layers and then copying into a queue multiplies the cost. Take the parameter by value and `std::move` it into storage. 2. **The copyability requirement.** `std::function` needs a copy-constructible target, so state owned through a `std::unique_ptr` is often converted to a `std::shared_ptr` just to fit, which adds a control-block allocation. 3. **The capture itself.** A captured `std::string` longer than the library's small-string capacity allocates when the lambda is built, before the wrapper is involved. ## Removing the allocation 1. **Capture less.** Capture a job id, an index into a pool or a pointer to long-lived state instead of strings and vectors. Two pointers fit both libraries' buffers; on libstdc++ the capture must also be trivially copyable. 2. **Avoid the erasure on the hot path.** If `submit` can be a template, `template <class F> void submit(F&& job)` keeps the closure's real type and can place it in storage the scheduler controls, with no erasure at all. 3. **Use a fixed-capacity job type.** Many engines and trading systems write a small type-erased callable with an inline buffer of known size and a `static_assert` that rejects bigger targets, turning a hidden allocation into a compile error. 4. **Store a reference when the callable outlives the wrapper.** `std::function<void()> f = std::ref(job);` stores a `std::reference_wrapper`; the standard says that construction throws nothing, and both libc++ and libstdc++ keep such a target inline. The caller must keep `job` alive. 5. **C++23 `std::move_only_function`.** It accepts move-only targets, so `std::unique_ptr` can stay a `std::unique_ptr`, and its call operator carries `const` and `noexcept` in the signature. The standard again only recommends a small-object optimization, which it notes can apply only to nothrow-move-constructible targets. libstdc++ in GCC 16 provides it; libc++ in LLVM 23 does not yet. ```cpp // C++23, not compiled here: libc++ in LLVM 23 lacks std::move_only_function std::move_only_function<void()> job = [buf = std::make_unique<Buffer>()] { buf->flush(); }; // move-only target accepted ``` ## Measuring before and after - Count allocations in a test by replacing `operator new`, as the example does, or use the platform's allocation profiler. - Check both standard libraries you ship on; a capture that is free on libc++ may allocate on libstdc++. - Re-measure after each change: fixing the wrapper can expose the string copies inside the capture as the next cost. ## How to answer Explain the inline buffer and that its size is an implementation choice, give the libc++ and libstdc++ conditions, list the hidden copies and the copyability requirement, then offer the fixes in order of intrusiveness.
- Does std::move_only_function guarantee no allocation for small targets?No. Like `std::function`, the C++23 standard only recommends avoiding dynamic allocation for a small contained value, and its note says that optimization can only apply to targets that are nothrow move-constructible. Buffer size and conditions remain implementation choices, so measure on each standard library you ship.
- Why does libstdc++ insist on trivially copyable targets for its inline buffer?libstdc++ treats a callable as storable inline only if it is location-invariant, which it defines as trivially copyable: such a target can be relocated by copying its bytes when the wrapper is moved or swapped, without calling constructors. A capture with a `std::string` is not trivially copyable, so libstdc++ always places it on the heap.
- The C++ standard guarantees that small lambdas never allocate inside std::function.
- Every standard library uses the same inline-buffer size for std::function.
- Moving a std::function into a queue copies its target and allocates again.
- std::move_only_function never allocates, because it cannot be copied.
- Capturing a std::string is free once the string fits its small-string buffer.
Assumes C++23 for std::move_only_function; the inline-buffer sizes are libc++ (LLVM 23, default ABI) and libstdc++ (GCC 16) choices, not the standard's.