Puma workers in a large legacy Rails marketplace grow from 400 MB to 1.2 GB daily on glibc malloc; why might the allocator be to blame, and what do jemalloc and MALLOC_ARENA_MAX=2 change?
answer
- RSS climbs while live objects stay flat
- per-thread malloc arenas fragment
- libjemalloc2 plus LD_PRELOAD
- set in the image's ENV, all processes
- MALLOC_ARENA_MAX=2 is the slower fallback
basics
~20 sWith several Puma threads per worker, glibc malloc spreads allocations over many arenas and fragments, so freed memory is not returned and RSS creeps. jemalloc, preloaded via LD_PRELOAD as Rails' Dockerfile does, fragments far less; MALLOC_ARENA_MAX=2 limits arenas but can slow Ruby.
solid answer
~40 sRuby keeps its objects in its own heap, but strings, arrays and other buffers are malloc'd. glibc's malloc gives concurrently allocating threads separate **arenas** to avoid lock contention, and in a multi-threaded Puma worker those arenas fragment: much of the memory Ruby frees cannot be handed back to the OS, so resident size climbs and plateaus high even though `GC.stat` shows a steady heap. The Rails tuning guide calls this out and recommends **jemalloc**. The Dockerfile Rails generates installs `libjemalloc2`, symlinks it to `/usr/local/lib/libjemalloc.so` and sets `LD_PRELOAD` in the base image's `ENV`, so every process in the container uses it. Without Docker you must install and preload it yourself. If you cannot, `MALLOC_ARENA_MAX=2` caps glibc's arenas and cuts fragmentation, at the risk of slowing Ruby.
code
dockerfile · 12 lines# Base stage of the Dockerfile Rails 8.1 generates (excerpt)
RUN apt-get update -qq && \
apt-get install --no-install-recommends -y curl libjemalloc2 libvips sqlite3 && \
ln -s /usr/lib/$(uname -m)-linux-gnu/libjemalloc.so.2 /usr/local/lib/libjemalloc.so && \
rm -rf /var/lib/apt/lists /var/cache/apt/archives
# Set production environment variables and enable jemalloc for reduced memory usage and latency.
ENV RAILS_ENV="production" \
BUNDLE_DEPLOYMENT="1" \
BUNDLE_PATH="/usr/local/bundle" \
BUNDLE_WITHOUT="development" \
LD_PRELOAD="/usr/local/lib/libjemalloc.so"go deeper
Recall that Rails' Dockerfile preloads jemalloc to reduce memory use, and that MALLOC_ARENA_MAX=2 is the fallback when jemalloc is unavailable.
Explain arenas and fragmentation, how LD_PRELOAD swaps the allocator for every process in the image, and the speed caveat of MALLOC_ARENA_MAX.
Separate fragmentation from leaks using RSS shape and GC.stat, verify the loaded allocator, and canary the change before fleet rollout.
Fold allocator choice into the per-worker memory budget that decides host size and worker density across the fleet.
## Two kinds of memory in a Ruby process A Rails worker's resident memory (**RSS**) has two very different parts: - the **Ruby object heap**, slots for objects managed by Ruby's garbage collector, which `GC.stat` describes; - **native allocations** made through the C allocator, `malloc`: string and array buffers, hash tables, C extension memory, YJIT's code. When RSS grows, the first question is which part grows. If the count of live heap slots keeps rising, something is retaining Ruby objects, which is a leak to hunt with Ruby's GC tooling. If the heap is flat while RSS climbs and then levels off, the native side is the suspect, and the allocator is the usual culprit. ## Why glibc malloc fragments under Puma On most Linux distributions `malloc` is glibc's. To keep threads from fighting over one lock, glibc lets threads that allocate at the same time use separate **arenas**, independent pools of memory. A Puma worker runs several request threads, so its allocations spread over several arenas. Over hours of mixed request sizes, each arena ends up with free gaps it cannot hand back to the operating system, because a few live allocations pin the pages around them. That is **fragmentation**: memory the program has freed but the process still holds. The Rails guide on tuning for deployment names this directly. Running Puma with multiple threads on the default Linux allocator can cause an unexpected increase in memory from fragmentation, which in turn can stop you from running enough workers to use every core. The marketplace's 400 MB to 1.2 GB curve that flattens out by evening fits this pattern. ## The fix Rails ships: jemalloc through LD_PRELOAD **jemalloc** is an alternative `malloc` implementation designed to keep fragmentation low in multi-threaded programs. Rails has put it in the generated Dockerfile since 7.2: 1. The base stage installs the `libjemalloc2` package with the other runtime packages. 2. It symlinks the architecture-specific `libjemalloc.so.2` to `/usr/local/lib/libjemalloc.so`, so the path is the same on amd64 and arm64. 3. The base image's `ENV` sets `LD_PRELOAD="/usr/local/lib/libjemalloc.so"`. `LD_PRELOAD` tells the dynamic linker to load that library before any other. Its `malloc` and `free` then take the place of glibc's for the whole process, with no Ruby or app change. Because it sits in the image's environment, it applies to every process the container runs: Puma, `bin/jobs`, consoles, one-off tasks. The Dockerfile's own comment sums up the intent: enable jemalloc for reduced memory usage and latency. Off Docker, nothing does this for you. Install jemalloc on the host and set `LD_PRELOAD` in the service definition, or use a Ruby built with its `--with-jemalloc` configure option. Verify rather than assume: on Linux, `grep jemalloc /proc/<pid>/maps` shows whether the library is mapped into a running worker. ## The fallback: MALLOC_ARENA_MAX=2 When jemalloc is not an option, the guide's fallback is `MALLOC_ARENA_MAX=2`, an environment variable glibc reads at startup. Fewer arenas means less fragmentation, but threads now share arenas and can contend on their locks, so the guide warns it may make Ruby slower and calls jemalloc the preferred solution. | Option | Fragmentation | Speed | How it is set | |---|---|---|---| | glibc defaults | high with many threads | baseline | nothing | | `MALLOC_ARENA_MAX=2` | lower | may be slower | env var for the process | | jemalloc via `LD_PRELOAD` | lowest of the three | the Dockerfile comment cites lower latency | package plus `LD_PRELOAD` | ## Diagnosing in order 1. Plot worker RSS over a day: steady growth without a ceiling suggests a leak; growth to a plateau suggests fragmentation. 2. Compare it with live heap slots from `GC.stat`: flat slots plus rising RSS points away from Ruby objects. 3. Confirm which allocator the process actually loaded. 4. Switch to jemalloc in one canary host and compare the curves before rolling out. ## Why it matters beyond the number Fragmentation is not only wasted RAM. Worker density is set by the worst steady-state RSS, so a worker that drifts to three times its boot size forces either fewer workers per host, leaving cores idle, or periodic restarts to reclaim memory. Restarting workers on a timer hides the symptom and adds boot cost and latency spikes. Fixing the allocator removes the reason for those restarts and lets the memory budget be set from steady-state numbers that hold all day. None of this replaces fixing a genuine leak. An allocator change hides nothing that is retained on purpose.
- After moving to jemalloc, worker memory still rises without levelling off; what next?Then the allocator was not the whole story. A curve with no plateau, together with rising live slots in `GC.stat`, means Ruby objects are being retained: a class-level cache, a memoised Hash keyed by user input, or a gem holding references. Hunt that with heap dumps and allocation tracing, which belongs to Ruby's GC tooling rather than to deployment settings.
- Why can fragmentation cap how many workers a box runs, not just waste memory?Worker count is limited by memory per worker. If each worker's RSS drifts from 400 MB to 1.2 GB, a box sized for eight workers at the start may fit only three by evening, leaving cores idle or forcing restarts. Lower steady-state memory per worker is what lets one worker per core actually fit.
- Memory that grows and then plateaus in Puma workers always means a Ruby object leak.
- jemalloc is switched on by a plugin line in config/puma.rb.
- MALLOC_ARENA_MAX=2 is faster than jemalloc and the Rails guide's first choice.
- LD_PRELOAD in the generated Dockerfile affects Puma only, not bin/jobs or a console.
- Setting config.yjit = false makes Ruby fall back to a less fragmenting allocator.
Assumes the Dockerfile Rails 8.1 generates; Rails added jemalloc to the generated Dockerfile in 7.2.