A Linux network service burns most of its CPU in kernel mode because it issues a very large number of small system calls. Why is each crossing of the user/kernel boundary expensive, and what changes make the same work cost fewer crossings?

## What the crossing actually costs The expensive part of a system call is rarely the work it names; it is the transition. On entry the CPU switches privilege level, the kernel switches to the thread's kernel stack, and the user-mode register state is saved so it can be restored exactly. The syscall number is bounds-checked and dispatched. Every pointer argument is treated as untrusted and copied through validating helpers, because the kernel must never dereference a user address directly. On exit the whole thing unwinds, plus a check for pending work such as signals or a reschedule request. Speculative-execution mitigations made this dearer. With kernel page-table isolation, entering the kernel switches to a different set of page tables and leaving switches back, which invalidates translation-buffer entries and costs real time; branch-predictor barriers add more. The upshot is that on a mitigated x86-64 machine a trivial system call costs on the order of hundreds of nanoseconds where an in-process function call costs single-digit nanoseconds. That ratio is the whole reason syscall-bound workloads exist. None of this is wasted on a call that transfers a megabyte. It is entirely wasted on a call that transfers sixteen bytes. ## Mode switch is not a context switch The two get conflated constantly, and the distinction matters for diagnosis. A **mode switch** changes privilege level within the same thread: same address space (mitigations aside), same scheduling entity, same caches mostly warm. A **context switch** replaces the running thread with a different one: the scheduler runs, register and floating-point state is swapped, and if the new thread belongs to another process the address space changes, so translation buffers and caches lose their working set. A context switch is typically an order of magnitude more disruptive than a mode switch, and its real cost is the cache damage afterwards rather than the switch itself. A system call causes a context switch only if it blocks — waiting on I/O, on a lock, on an empty socket. A non-blocking call that is satisfied immediately, such as a `read()` served from the page cache, is a mode switch and nothing more. So "high kernel-mode CPU time" and "high context-switch rate" are different symptoms with different remedies: the first says you are calling too often or doing too much per call, the second says your threads keep going to sleep. ## Making the same work cost fewer crossings Every technique here is a form of amortisation — the same bytes, fewer transitions. **Buffer in user space.** The classic case is writing a log line or a protocol message at a time. Accumulating output and flushing when the buffer fills turns thousands of `write()` calls into a handful. This is exactly what stdio does, and it is why a program that writes with `printf()` performs so differently from one that writes each line with an unbuffered `write()`. **Vectored I/O.** When the pieces cannot be concatenated cheaply — a status line, headers and a body held in separate allocations — `writev()` takes an array of buffers and writes them in one call. `readv()` is the mirror image. One transition, several buffers, no memory copying to join them. **Zero-copy transfers.** Serving a file over a socket the naive way costs a `read()` into user memory and a `write()` out of it: two crossings and two copies of every byte. `sendfile()` moves the data inside the kernel with one call and no user-space copy; `splice()` generalises that to pipes. For a static-content server this is the difference between being CPU-bound and being network-bound. **Batch the event loop.** A readiness API that returns many ready descriptors from a single call amortises the crossing across all of them. `epoll_wait()` fills an array of events per call; the pathological alternative is asking about descriptors individually in a loop. Combine that with reading until the socket is drained rather than one small chunk per wakeup. **Submission-queue I/O.** `io_uring` takes the idea further: the application and the kernel share ring buffers in memory, so many operations can be submitted and completed with few or no transitions at all. It is the right tool when you have already exhausted batching and the syscall rate is still the wall you are hitting. **Do not call at all when you need not.** Reading the clock is the common offender, and it is already handled by the kernel mapping timekeeping data into every process. But application-level equivalents exist: caching a value your code re-derives from the kernel on every request often removes an entire class of calls. ## Where this reasoning stops Syscall overhead is a real ceiling, but it is not the explanation for every slow service. If the calls are blocking on disk or network, the transitions are noise next to the wait, and batching buys nothing; the fix is concurrency or fewer round trips. The question to answer first is whether kernel time is being spent *entering* the kernel or *inside* it doing genuine work — high kernel CPU with a huge call count and small payloads points at the boundary, while high kernel CPU with a modest call count points at the work itself, such as checksum, copy or lock contention in the network stack.