Why would a Node.js service hashing passwords with crypto.pbkdf2() see outbound calls by hostname and file reads slow down during login bursts while event loop delay stays low, and how would you fix it?
answer
- the main thread is not the bottleneck
- one shared queue, four slots
- getaddrinfo waits behind the hashes
- size fixed before startup
- take hot paths off the pool
basics
~20 scrypto.pbkdf2(), dns.lookup() and async fs calls share libuv's threadpool, four threads by default, so a burst of hashes makes lookups and reads queue behind them. Size UV_THREADPOOL_SIZE before start, keep dns.lookup() off hot paths, and cap or isolate hashing.
solid answer
~50 sEvent loop delay looks healthy because the main thread is idle; the queue is in libuv's threadpool. `crypto.pbkdf2()` runs there, and so do `dns.lookup()`, which `http.request()` and `net.connect()` call for each new connection to a hostname, and every async `fs` call. With the default 4 threads, a login burst fills every slot with hashes, and unrelated lookups and reads wait in the same queue. To confirm it, time a cheap `fs.stat()` or `dns.lookup()` from call to callback under load, then repeat with a larger pool. Fixes: set `UV_THREADPOOL_SIZE` in the environment before Node starts, since assigning `process.env` at run time is not guaranteed to work; cut lookups with keep-alive reuse, IP addresses or a caching `lookup`; and bound hashing concurrency, or run it with `crypto.pbkdf2Sync()` inside a dedicated `worker_threads` pool so it cannot crowd out I/O.
code
bash · 1 lineUV_THREADPOOL_SIZE=16 node server.jsgo deeper
Recall that crypto.pbkdf2(), dns.lookup() and async fs calls all share one small libuv threadpool of four threads by default.
Explain why unrelated calls slow down together: one shared queue, hashes holding threads for their whole duration, and dns.lookup() hidden inside new outbound connections.
Demonstrate the diagnosis and the ordered fix: healthy loop delay, a timed probe, removing avoidable lookups, isolating hashing, then sizing the pool before startup.
Frame pool sizing as a budget shared by every library in the process, and set a team default for where CPU-heavy crypto runs and how the pool size is configured.
## The symptom pattern Threadpool saturation looks different from a blocked main thread, and telling the two apart is most of the diagnosis: - **Event loop delay stays low.** The main thread is mostly idle, waiting in the poll phase, so loop-delay and loop-utilisation metrics (covered in the lesson on blocking and lag measurement) look healthy. - **Unrelated operations slow down together.** Outbound HTTP calls to hostnames, file reads, template loads and anything else that touches the pool get slower at the same moment. - **The slowdown tracks one workload.** Here it rises with the login rate, because each login is one `crypto.pbkdf2()` task. - **Calls by IP address and plain socket traffic stay fast**, because neither touches the pool. ## Why it happens 1. libuv's **threadpool** is one queue served by `UV_THREADPOOL_SIZE` threads, **4** by default, shared by the whole process. 2. `crypto.pbkdf2()` holds a pool thread for the entire key derivation, often tens to hundreds of milliseconds. 3. `dns.lookup()` is implemented as a blocking `getaddrinfo(3)` call on a pool thread, and `http.request()` and `net.connect()` use it by default to resolve a hostname for each **new** connection. 4. Every callback and promise `fs` call also runs on the pool. 5. During a burst, the queue fills with hashes. A lookup or file read submitted next waits until a hash finishes, so its latency becomes "queue wait plus its own time", while the main thread has nothing to do. ## Confirming it - Time a cheap probe from call to callback under load, such as `fs.stat()` on a small file or `dns.lookup('localhost')`. A few milliseconds idle and hundreds under a login burst points at the pool. - Restart once with a larger `UV_THREADPOOL_SIZE` in a test environment and repeat the load. If probe latency drops sharply, the pool was the bottleneck. - Look at per-thread CPU in the operating system: the libuv bundled with Node 24.21.0 (1.52) names its pool threads `libuv-worker`, a default added in libuv 1.50.0; all of them sitting near 100% during the burst is a strong sign. - Record the time between starting a request and the socket's `'lookup'` event, which `net` emits after resolving the host name and before connecting. ## Fixes and their trade-offs | Fix | What it changes | Cost | |---|---|---| | Raise `UV_THREADPOOL_SIZE` at startup | More slots, so cheap tasks find a free thread beside the hashes | Memory per thread; no faster hashing beyond the free cores | | Reuse connections with a keep-alive agent | Fewer new connections, so fewer `dns.lookup()` calls | Pooled sockets to manage, covered with the HTTP client material | | Pass a custom `lookup` to `http.request()` or `net.connect()` | Lets you cache results or use `dns.resolve*()`, which does not use the pool | You own caching, TTLs and `/etc/hosts` differences | | Connect by IP address where you control the target | `net` skips the lookup entirely | Loses DNS-based failover | | Bound hashing concurrency | Leaves some pool threads free for I/O | Logins queue in your code instead | | Hash with `crypto.pbkdf2Sync()` inside a fixed `worker_threads` pool | Hashing gets its own threads and limit, off the shared pool | Message passing and one isolate per worker | The last row has a trap: an async `crypto.pbkdf2()` called **inside** a worker still lands on the process-wide libuv pool, because every event loop in the process shares it. Only the synchronous call keeps the work on the worker's own thread. ## Sizing UV_THREADPOOL_SIZE - Set it in the **environment that launches Node** (the shell command, the service unit, the container definition). Node's documentation warns that assigning `process.env.UV_THREADPOOL_SIZE` from JavaScript is not guaranteed to work, because the pool may already exist by the time user code runs. - The maximum is **1024** in current libuv. On first use libuv creates every configured thread, each with an 8 MB stack that most platforms grow lazily, so a large value costs memory even when idle. - CPU-bound tasks such as `crypto.pbkdf2()` cannot run faster than the cores available; past that point extra threads only time-slice, and the main thread competes for the same cores. - Waiting tasks such as a slow `getaddrinfo` or a slow disk benefit most from a larger pool, because their threads are idle while they wait. A sound sequence is: remove avoidable lookups first, then give CPU-heavy hashing its own bounded home, then size the pool for the I/O that remains, and keep a latency probe in place so the next saturation is visible.
- Why is setting `process.env.UV_THREADPOOL_SIZE` at the top of the entry file unreliable?Node's CLI documentation says the threadpool may already have been created during runtime initialisation, before any user code runs, so a value assigned from JavaScript is not guaranteed to take effect. Put it in the environment of whatever launches the process instead: the shell command, the service definition or the container's environment.
- Would raising `UV_THREADPOOL_SIZE` to 128 make each `crypto.pbkdf2()` call finish faster?No. Key derivation is CPU-bound, so once pool threads outnumber the free cores, extra threads only share the same cores and each hash takes longer. A larger pool helps cheap or waiting tasks find a free slot beside the hashes. libuv also creates every configured thread on first use, so a large value costs memory.
- Why does moving hashing into `worker_threads` help only if the worker calls `crypto.pbkdf2Sync()`?Each worker has its own event loop but shares the process-wide libuv threadpool, so an async `crypto.pbkdf2()` called in a worker still queues on that pool. Calling `crypto.pbkdf2Sync()` inside a fixed set of workers keeps the computation on the workers' own threads, gives hashing its own concurrency limit, and frees the pool for `fs` and `dns.lookup()`.
A branch with four tellers and one greeter at the door: the greeter (the main thread) is idle and the lobby looks calm, but four customers each opening a mortgage occupy every teller, so someone with a thirty-second deposit (a dns.lookup() or fs.stat()) waits behind them. Hiring a greeter does not help; more tellers, or a separate desk for mortgages, does.
- Event loop delay is low, so the Node process cannot be the bottleneck.
- Assigning process.env.UV_THREADPOOL_SIZE at run time reliably resizes the pool.
- dns.lookup() is network I/O, so it never uses the threadpool.
- Calling crypto.pbkdf2() inside a worker thread gives it a private threadpool.
- A bigger threadpool always makes CPU-bound hashing finish faster.