As lead of a C++ market-data handler, how would you set an allocation budget and allocator policy for the tick path, and how would you enforce them?
answer
- steady state versus startup
- one strategy per data lifetime
- what happens on overflow
- a test that counts allocations
- measure fallbacks in production
basics
~20 sSet the budget at zero heap allocations per tick after warm-up, give each data lifetime its allocator (startup reserves, pools, a per-tick arena), choose throw or counted fallback on overflow, and enforce it with an allocation-counting test and production counters.
solid answer
~50 sI would make the budget explicit: **zero heap allocations per tick in steady state**, with startup and rare resync paths exempt and named. The policy then follows data lifetimes: long-lived state such as order books and symbol tables is sized at startup with `reserve` and pools; per-tick scratch lives in a `std::pmr::monotonic_buffer_resource` per thread, reset every tick; hot-path code avoids `std::function` with large captures, `std::make_shared`, string formatting and node-based maps. The key judgment is overflow: an upstream of `std::pmr::null_memory_resource()` makes a miss throw, which suits tests; production may prefer a counting fallback to the heap that raises an alert rather than dropping a tick. Enforcement is a CI test that replaces `operator new`, replays recorded ticks and asserts zero after warm-up, plus a fallback counter in production. The costs are reserved memory per instance, a virtual call per pmr allocation and stricter code review.
code
cpp · 30 lines#include <array>
#include <atomic>
#include <cstddef>
#include <cstdio>
#include <memory_resource>
#include <vector>
// Upstream for the tick resource: serve overflow from the heap, but count it so monitoring can alert.
class CountingFallback : public std::pmr::memory_resource {
public:
std::atomic<long> fallbacks{0};
private:
void* do_allocate(std::size_t bytes, std::size_t align) override {
fallbacks.fetch_add(1, std::memory_order_relaxed);
return std::pmr::new_delete_resource()->allocate(bytes, align);
}
void do_deallocate(void* p, std::size_t bytes, std::size_t align) override {
std::pmr::new_delete_resource()->deallocate(p, bytes, align);
}
bool do_is_equal(const memory_resource& other) const noexcept override { return this == &other; }
};
int main() {
alignas(std::max_align_t) static std::array<std::byte, 4096> scratch;
CountingFallback fallback;
for (int tick = 0; tick < 3; ++tick) {
std::pmr::monotonic_buffer_resource mr(scratch.data(), scratch.size(), &fallback);
std::pmr::vector<double> prices(&mr);
prices.resize(tick == 2 ? 1000 : 100); // tick 2 is a burst: 8000 bytes do not fit
std::printf("tick %d: fallbacks so far = %ld\n", tick, fallback.fallbacks.load());
}
}go deeper
Know that a hot path can have an allocation rule, and that reserving and reusing memory at startup is how such a rule is met.
Explain how a counting operator new test proves zero allocations after warm-up, and why it must run on the toolchain you ship.
Map each data lifetime to an allocator, design the null-upstream test mode and the counting fallback, and name the counter's blind spots.
Own the trade-offs: memory per instance, pmr versus template allocators, throw versus fall back on overflow, and whether zero is the right number for this latency target.
## Why the budget has to be written down "Avoid allocations in the hot path" is advice; a budget is a number someone can test. Tick-to-trade latency is judged on its tail, and a heap allocation is one of the commonest sources of tail spikes: an allocator lock, a page fault on fresh memory, an occasional request to the operating system. A lead's job is to turn that into a rule with three parts: **what is counted**, **what is allowed**, and **what happens when the rule is broken**. A workable budget for a market-data handler: - **Zero heap allocations per tick in steady state**, measured on the tick thread after a warm-up period. - **Startup is exempt**: building tables, reserving vectors and filling pools may allocate freely before the first subscription. - **Named exceptions**, each with an owner: a full book resync after a gap, an administrative command, logging on an error path. ## Policy: one allocation strategy per data lifetime | Data | Lifetime | Strategy | |---|---|---| | Order books, symbol tables, connections | the whole session | sized at startup: `reserve`, fixed arrays indexed by symbol id, pools for book levels | | Book levels and orders that come and go | individual, reused | a pool: `std::pmr::unsynchronized_pool_resource` per thread, or a custom fixed-block pool | | Parsed fields, temporary vectors, scratch strings | one tick | `std::pmr::monotonic_buffer_resource` over a per-thread buffer, reset every tick | | Messages handed to another thread | until consumed | preallocated slots in a ring buffer, never `std::make_shared` per message | Two mechanism choices sit under the table, and both are trade-offs rather than right answers: 1. **`std::pmr` or template allocators.** `std::pmr` keeps one container type whatever the resource, so interfaces stay simple, at the cost of a virtual call per allocation and a resource pointer in every container. A custom allocator as a template argument can inline completely, but it changes the container's type, and that type spreads through every function signature that takes the container. 2. **Overflow behaviour.** An upstream of `std::pmr::null_memory_resource()` throws `std::bad_alloc`, which is perfect in tests: a budget miss fails loudly. In production, throwing on the tick path may drop market data; many teams prefer a fallback upstream that delegates to `new_delete_resource()` and counts every use, so monitoring raises an alert while trading continues. The example shows such a counting upstream catching one burst tick. Also part of the policy: which types are banned in tick code. A short list reviewers can check — `std::function` holding more than a pointer, `std::make_shared`, `std::to_string` and `std::ostringstream`, `std::map` and `std::unordered_map` inserts, `throw` on the normal path — beats a vague instruction to be careful. ## Enforcement A budget nobody measures decays within a few releases. Three layers keep it honest: - **A CI test** that links a counting global `operator new`, warms the handler up, replays a recorded session and asserts the count did not move. Run it on the toolchain you ship, because small-buffer sizes in `std::string` and `std::function` differ between libc++ and libstdc++. - **A strict test mode** that sets the default memory resource to `null_memory_resource()`, so any pmr container built without an explicit resource throws instead of quietly using the heap. `std::pmr::set_default_resource` changes one program-wide pointer, not a per-thread one, so this belongs in a test process rather than in production. - **Production counters**: the fallback upstream's count, exported as a metric, plus a heap profiler in staging for allocations the counter cannot see, such as a third-party C library calling `malloc` directly. ## What the policy costs - **Memory.** Buffers and pools are sized for the peak plus headroom, per thread and per instance; across many instances that is real money. - **Complexity.** `std::pmr::string` and `std::string` are different types; converting at an interface boundary allocates, so the policy has to say where the boundary is. - **Velocity.** Every new feature on the tick path pays for review against the banned list and for keeping the replay test green. - **Third-party code.** A parser or compression library that allocates internally either gets wrapped, replaced, or written down as a named exception with its measured cost. The judgment a lead brings is where to draw these lines for this engine: a team chasing single-digit microseconds may accept the memory and the review cost, while a team whose latency target is milliseconds may budget a handful of allocations per tick and spend the effort elsewhere.
- How would you size the per-tick scratch buffer, and what do you do when the market outgrows it?Measure the high-water mark of the monotonic resource over busy recorded sessions, such as an open or a volatile close, and add headroom, often a factor of two. Export the fallback counter: a rising count means the market outgrew the estimate, and the fix is a larger buffer in the next release, not an emergency change to the tick code.
- When would you accept a small non-zero allocation budget per tick instead of zero?When the latency target leaves room for it and zero would cost more than it saves: for example a handler measured in milliseconds, or a path that depends on a third-party library you cannot change. Then the budget becomes a measured ceiling with an alert, so a regression from two allocations to two hundred is still caught.
- Calling set_default_resource on the tick thread makes the change for that thread only
- A zero-allocation rule needs no exceptions, startup included
- Template allocators are always better because they avoid pmr's virtual call
- A global operator new counter proves no allocation of any kind happens
- Throwing on the first budget miss is the only safe production behaviour