How would you choose memory and CPU limits for a latency-sensitive containerized service, given that exceeding the memory limit kills the process while exceeding the CPU quota only delays it?

## Start from the failure modes Memory and CPU limits are both cgroup settings, but they fail in opposite ways. Crossing memory.max triggers reclaim and then a SIGKILL: an abrupt, undiagnosable termination, dropped in-flight requests, and a restart. Crossing the CPU quota causes throttling: the workload keeps running, just slower, and recovers on its own when demand falls. One is a cliff, the other is a slope. Sizing policy should reflect that asymmetry rather than applying one formula to both. ## Sizing memory Measure the real footprint under realistic load, including warmup, peak traffic, and the tail-risk operations that only happen occasionally - a large export, a cache rebuild, a retry storm inflating in-flight buffers. The number that matters is the non-reclaimable part: the anon component from memory.stat and the high-water mark from memory.peak. Page cache in memory.current is reclaimable and should not drive the limit. Add headroom, commonly 25-50% over observed peak, because the cost of being wrong is asymmetric: a slightly generous limit wastes some node capacity, a slightly tight one kills the process during exactly the traffic spike you cared about. Then make the failure diagnosable. Cap the runtime's own allocation *below* the container limit - a JVM with MaxRAMPercentage in the 65-75% range, Node with max-old-space-size, Go with GOMEMLIMIT - so the process hits an application-level memory error, logs it, dumps state, and exits in a recognizable way instead of vanishing under SIGKILL. Account for everything the container limit covers: heap plus metaspace, thread stacks (stack size times thread count is not negligible at high concurrency), code cache, direct and mapped buffers, native allocator overhead and any helper processes sharing the cgroup. ## Sizing CPU The key insight is that throttling is driven by burstiness against a fixed 100 ms period, not by average utilization. A limit equal to mean usage guarantees throttling on every spike. For latency-sensitive services, either set the quota comfortably above the burst rate, or drop the hard quota in favour of a weight so the service can borrow idle capacity while still being protected under contention. Where the platform supports it, burst tolerance (cpu.max.burst) lets unused quota absorb short spikes. Whatever the quota, align concurrency to it: thread pools, GC parallelism, connection pools and parallel-stream defaults sized from host core count will drain a small quota instantly and turn a burst into a multi-tens-of-milliseconds stall. This is often a bigger win than raising the limit. ## Choosing a posture - **Hard limits everywhere** maximize predictability and protect neighbours, at the cost of utilization and the risk of self-inflicted throttling and kills. - **Weights plus generous memory limits** maximize utilization and latency, but a misbehaving workload can degrade its neighbours' CPU, and a node-level memory shortfall becomes possible. A common compromise: hard memory limits always (memory is not shareable in a graceful way, and the node-level OOM killer is worse than a container-level one), CPU quotas generous or replaced by weights for latency-critical tiers, strict quotas for batch and untrusted work. ## Density versus blast radius Limits are also a packing decision. Tight limits fit more work per node and increase the chance that a normal spike causes a kill; generous limits waste capacity but degrade gracefully. Ask what the service's error budget can absorb: a user-facing API justifies slack that a nightly batch job does not. ## Close the loop Sizing is not a one-time calculation. Load-test to find the peak footprint and the throttling knee. Alert on the memory.events oom_kill counter (which also catches kills of non-PID-1 children that leave a container silently degraded), on memory.current approaching memory.max, and on the nr_throttled to nr_periods ratio. Re-derive limits after dependency upgrades, traffic growth or concurrency changes, and record why each number was chosen so the next engineer does not treat it as arbitrary. In orchestrated environments these numbers become the platform's requests and limits, but the reasoning above is what produces them.