For a Rails 8.1 app on Puma, how do you choose processes per CPU core and threads per process when optimizing for throughput versus latency?
answer
- processes for parallelism, threads for I/O
- default 3 threads since Rails 7.2
- throughput: one process per core
- latency: 1 thread, 1.3-1.5 processes per core
- RAM per core can cap processes
basics
~20 sProcesses give parallelism and threads only overlap I/O. For throughput, run one process per core and raise threads until latency suffers; for latency, keep threads low, even one per process with about 1.3 to 1.5 processes per core.
solid answer
~50 sOn CRuby each process runs Ruby on one core at a time, so **processes** are the parallelism and **threads** only fill the time a request spends waiting on I/O. Rails' tuning guide gives two directions. For **throughput**, run one process per core and add threads per process until the latency cost is no longer acceptable. For **latency**, keep threads low; at the extreme, one thread per process and 1.3 to 1.5 processes per core, so a core is not idle while one process waits. Rails 7.2 lowered the generated default from 5 threads to 3. A well-tuned app spends no more than about half its time in I/O, so extra threads mostly queue for the lock; apps making slow third-party calls inline may justify more. Memory decides the rest: if RAM per core cannot hold that many processes, the host shape, not the formula, is the limit. Then load-test.
go deeper
Recall that processes give parallelism on CRuby, threads help while requests wait on I/O, and Rails generates 3 threads per process.
Explain why threads beyond the app's I/O share hurt latency, and why Rails 7.2 cut the default from 5 to 3.
Size per core for a stated goal, account for memory per core and copy-on-write, and validate each change with load tests and p95 latency.
Choose the goal itself: cost-per-request versus user latency, and the host shapes that make either one achievable across the fleet.
## Two dials, two different resources A Rails app on Puma has two concurrency dials, and they buy different things: - **Processes** (Puma workers) give **parallelism**. Separate CRuby processes can execute Ruby on separate cores at the same moment. - **Threads** inside one process give **concurrency for I/O**. While one thread waits for the database or an HTTP response, another can run Ruby. But only one thread per process executes Ruby at any instant, so a process never uses more than about one core. That asymmetry is why sizing is a trade between three properties the Rails guide names: **throughput** (requests per second the server sustains), **latency** (how long each request takes) and **memory**. Improving one usually hurts another. ## Why more threads can hurt latency When a thread's I/O completes it must wait for the process's lock before it can continue running Ruby. With many threads busy, a request that is ready to proceed waits behind others. The guide's rule of thumb: if the app spends about 50% of its time in I/O, more than 2 or 3 threads per process may severely hurt latency, and throughput gains flatten fast. A well-crafted Rails app without slow SQL or N+1 problems rarely exceeds that 50%, which is the reasoning behind the generated default. **Rails 7.2 changed the generated `config/puma.rb` from 5 threads to 3** for this reason. The release notes describe threads spending significant time waiting for the GVL when the count is too high, and settle on 3 as a balance. Apps that call third-party APIs inline, with much higher I/O share, are the exception that may benefit from more. ## The two recipes | Goal | Processes | Threads per process | Why | |---|---|---|---| | Throughput / server utilization | one per CPU core | raise from 3 until latency is unacceptable | every core busy, threads soak up I/O waits | | Latency | about 1.3 to 1.5 per core | low, down to 1 | no queueing for the lock; spare processes cover I/O waits | On the marketplace's 8-core app servers that means, for example: 1. **Throughput-leaning:** 8 workers with 3 threads, then load-test 4 or 5 threads and stop when p95 latency degrades. 2. **Latency-leaning:** 10 to 12 single-threaded workers, accepting the extra memory of more processes. The guide notes that "one per core" counts what the application may use. On a box shared with other services, count the cores you want this app to have. ## Memory decides what is possible Processes cost far more memory than threads, because forked workers share pages only partially through copy-on-write. Threads are cheaper but not free. So the recipes run into the hardware: - If RAM per core is too low to hold one worker per core, you cannot use all the CPU you pay for. Pick a plan with more memory per core, or cut memory per worker first (allocator, YJIT limits). - Larger hosts are more memory-efficient. The guide's example: 2 servers with 32 processes each use less memory per core than 16 servers with 4 processes each, because each server's master and shared pages are amortized over more workers. - Memory per worker is measured after warm-up under real traffic, not at boot. ## Where the guide's numbers come from The 1.3 to 1.5 figure is not magic. A single-threaded process that spends, say, a quarter of its time waiting on I/O leaves its core idle for that quarter. Running a few more processes than cores lets another process use the core during those waits, without any thread contention inside a process. The exact ratio depends on the I/O share, which is why it is a range. The throughput recipe works the other way round: threads fill the waits inside each process, and the process count stays at one per core. ## Measure, then decide The guide states plainly that no default is best for everyone. The process: - Fix the goal first: a throughput target at acceptable p95, or the lowest latency at acceptable cost. - Load-test with production-like requests after warm-up, changing one dial at a time. - Watch CPU per core, request queue time, p50/p95 latency and RSS per worker together. - Re-check after big changes in I/O share, such as moving slow API calls into jobs, which makes fewer threads optimal. Applying the numbers is a matter of `WEB_CONCURRENCY` and `RAILS_MAX_THREADS` in the deploy environment, and keeping the database pool at least as large as the thread count.
- The marketplace moves its slow payment-provider calls out of requests into background jobs; how should Puma's threads change?Requests now spend less of their time waiting on I/O, so extra threads have less waiting to overlap and more contention to cause. The case for more than 3 threads weakens, and fewer threads per process with the same or more processes per core usually gives better latency. Re-run the load test to confirm rather than assume.
- Why can two large servers serve the same traffic with less total memory than many small ones?Each server pays a fixed cost: a Puma master and pages that preloaded workers share. With 32 workers on a server, those costs are spread over 32 processes; with 4, over just 4. The Rails guide's example is 2 servers of 32 processes using less memory per core than 16 servers of 4.
- On CRuby, more threads per process always means more throughput with no latency cost.
- With one thread per process you should still run exactly one process per core.
- Rails' 3-thread default simply copies Puma's own default thread count.
- Splitting the same processes across many smaller servers saves memory.
- Threads are free, so only the number of processes affects memory.
Assumes Rails 8.1, whose generated config/puma.rb defaults to 3 threads per process; Rails 7.2 lowered it from 5.