skip to content

As lead of a C++ market-data handler, how would you set an allocation budget and allocator policy for the tick path, and how would you enforce them?

level: principalimportance: should knowfreq 22%

answer

  1. steady state versus startup
  2. one strategy per data lifetime
  3. what happens on overflow
  4. a test that counts allocations
  5. measure fallbacks in production

basics

~20 s

Set the budget at zero heap allocations per tick after warm-up, give each data lifetime its allocator (startup reserves, pools, a per-tick arena), choose throw or counted fallback on overflow, and enforce it with an allocation-counting test and production counters.

solid answer

~50 s

I would make the budget explicit: **zero heap allocations per tick in steady state**, with startup and rare resync paths exempt and named. The policy then follows data lifetimes: long-lived state such as order books and symbol tables is sized at startup with `reserve` and pools; per-tick scratch lives in a `std::pmr::monotonic_buffer_resource` per thread, reset every tick; hot-path code avoids `std::function` with large captures, `std::make_shared`, string formatting and node-based maps. The key judgment is overflow: an upstream of `std::pmr::null_memory_resource()` makes a miss throw, which suits tests; production may prefer a counting fallback to the heap that raises an alert rather than dropping a tick. Enforcement is a CI test that replaces `operator new`, replays recorded ticks and asserts zero after warm-up, plus a fallback counter in production. The costs are reserved memory per instance, a virtual call per pmr allocation and stricter code review.

code

cpp · 30 lines
cpp
#include <array>
#include <atomic>
#include <cstddef>
#include <cstdio>
#include <memory_resource>
#include <vector>
// Upstream for the tick resource: serve overflow from the heap, but count it so monitoring can alert.
class CountingFallback : public std::pmr::memory_resource {
public:
    std::atomic<long> fallbacks{0};
private:
    void* do_allocate(std::size_t bytes, std::size_t align) override {
        fallbacks.fetch_add(1, std::memory_order_relaxed);
        return std::pmr::new_delete_resource()->allocate(bytes, align);
    }
    void do_deallocate(void* p, std::size_t bytes, std::size_t align) override {
        std::pmr::new_delete_resource()->deallocate(p, bytes, align);
    }
    bool do_is_equal(const memory_resource& other) const noexcept override { return this == &other; }
};
int main() {
    alignas(std::max_align_t) static std::array<std::byte, 4096> scratch;
    CountingFallback fallback;
    for (int tick = 0; tick < 3; ++tick) {
        std::pmr::monotonic_buffer_resource mr(scratch.data(), scratch.size(), &fallback);
        std::pmr::vector<double> prices(&mr);
        prices.resize(tick == 2 ? 1000 : 100);  // tick 2 is a burst: 8000 bytes do not fit
        std::printf("tick %d: fallbacks so far = %ld\n", tick, fallback.fallbacks.load());
    }
}

go deeper

for a junior

Know that a hot path can have an allocation rule, and that reserving and reusing memory at startup is how such a rule is met.

for a middle

Explain how a counting operator new test proves zero allocations after warm-up, and why it must run on the toolchain you ship.

for a senior

Map each data lifetime to an allocator, design the null-upstream test mode and the counting fallback, and name the counter's blind spots.

for a principal

Own the trade-offs: memory per instance, pmr versus template allocators, throw versus fall back on overflow, and whether zero is the right number for this latency target.

## Why the budget has to be written down "Avoid allocations in the hot path" is advice; a budget is a number someone can test. Tick-to-trade latency is judged on its tail, and a heap allocation is one of the commonest sources of tail spikes: an allocator lock, a page fault on fresh memory, an occasional request to the operating system. A lead's job is to turn that into a rule with three parts: **what is counted**, **what is allowed**, and **what happens when the rule is broken**. A workable budget for a market-data handler: - **Zero heap allocations per tick in steady state**, measured on the tick thread after a warm-up period. - **Startup is exempt**: building tables, reserving vectors and filling pools may allocate freely before the first subscription. - **Named exceptions**, each with an owner: a full book resync after a gap, an administrative command, logging on an error path. ## Policy: one allocation strategy per data lifetime | Data | Lifetime | Strategy | |---|---|---| | Order books, symbol tables, connections | the whole session | sized at startup: `reserve`, fixed arrays indexed by symbol id, pools for book levels | | Book levels and orders that come and go | individual, reused | a pool: `std::pmr::unsynchronized_pool_resource` per thread, or a custom fixed-block pool | | Parsed fields, temporary vectors, scratch strings | one tick | `std::pmr::monotonic_buffer_resource` over a per-thread buffer, reset every tick | | Messages handed to another thread | until consumed | preallocated slots in a ring buffer, never `std::make_shared` per message | Two mechanism choices sit under the table, and both are trade-offs rather than right answers: 1. **`std::pmr` or template allocators.** `std::pmr` keeps one container type whatever the resource, so interfaces stay simple, at the cost of a virtual call per allocation and a resource pointer in every container. A custom allocator as a template argument can inline completely, but it changes the container's type, and that type spreads through every function signature that takes the container. 2. **Overflow behaviour.** An upstream of `std::pmr::null_memory_resource()` throws `std::bad_alloc`, which is perfect in tests: a budget miss fails loudly. In production, throwing on the tick path may drop market data; many teams prefer a fallback upstream that delegates to `new_delete_resource()` and counts every use, so monitoring raises an alert while trading continues. The example shows such a counting upstream catching one burst tick. Also part of the policy: which types are banned in tick code. A short list reviewers can check — `std::function` holding more than a pointer, `std::make_shared`, `std::to_string` and `std::ostringstream`, `std::map` and `std::unordered_map` inserts, `throw` on the normal path — beats a vague instruction to be careful. ## Enforcement A budget nobody measures decays within a few releases. Three layers keep it honest: - **A CI test** that links a counting global `operator new`, warms the handler up, replays a recorded session and asserts the count did not move. Run it on the toolchain you ship, because small-buffer sizes in `std::string` and `std::function` differ between libc++ and libstdc++. - **A strict test mode** that sets the default memory resource to `null_memory_resource()`, so any pmr container built without an explicit resource throws instead of quietly using the heap. `std::pmr::set_default_resource` changes one program-wide pointer, not a per-thread one, so this belongs in a test process rather than in production. - **Production counters**: the fallback upstream's count, exported as a metric, plus a heap profiler in staging for allocations the counter cannot see, such as a third-party C library calling `malloc` directly. ## What the policy costs - **Memory.** Buffers and pools are sized for the peak plus headroom, per thread and per instance; across many instances that is real money. - **Complexity.** `std::pmr::string` and `std::string` are different types; converting at an interface boundary allocates, so the policy has to say where the boundary is. - **Velocity.** Every new feature on the tick path pays for review against the banned list and for keeping the replay test green. - **Third-party code.** A parser or compression library that allocates internally either gets wrapped, replaced, or written down as a named exception with its measured cost. The judgment a lead brings is where to draw these lines for this engine: a team chasing single-digit microseconds may accept the memory and the review cost, while a team whose latency target is milliseconds may budget a handful of allocations per tick and spend the effort elsewhere.

  • How would you size the per-tick scratch buffer, and what do you do when the market outgrows it?
    Measure the high-water mark of the monotonic resource over busy recorded sessions, such as an open or a volatile close, and add headroom, often a factor of two. Export the fallback counter: a rising count means the market outgrew the estimate, and the fix is a larger buffer in the next release, not an emergency change to the tick code.
  • When would you accept a small non-zero allocation budget per tick instead of zero?
    When the latency target leaves room for it and zero would cost more than it saves: for example a handler measured in milliseconds, or a path that depends on a third-party library you cannot change. Then the budget becomes a measured ceiling with an alert, so a regression from two allocations to two hundred is still caught.

  • Calling set_default_resource on the tick thread makes the change for that thread only
  • A zero-allocation rule needs no exceptions, startup included
  • Template allocators are always better because they avoid pmr's virtual call
  • A global operator new counter proves no allocation of any kind happens
  • Throwing on the first budget miss is the only safe production behaviour