skip to content

In a Python profile, what separates wall-clock sampling from CPU-time sampling?

level: middleimportance: should knowfreq 40%

answer

  1. What triggers the next sample?
  2. Does waiting count as time?
  3. Latency question versus saturation question
  4. Blocked threads vanish from one of them
  5. Elapsed timer versus CPU-charged timer

basics

~10 s

Wall-clock sampling fires on elapsed time, so blocked threads keep accumulating samples and waiting shows up. CPU-time sampling fires in proportion to processor time consumed, so a thread parked on I/O contributes nothing.

solid answer

~50 s

The difference is what drives the interrupt. A wall-clock sampler wakes on elapsed time — a background thread sleeping on an interval, or an external process polling — so a thread blocked in a socket read, a database call or a lock acquisition still gets sampled, and the wait dominates the profile. A CPU-time sampler fires in proportion to processor time actually burned, for example via `signal.setitimer` with `signal.ITIMER_PROF`, so blocked time simply produces no samples. Use wall-clock when the complaint is latency: you want to see where the request is *waiting*. Use CPU time when the complaint is throughput or a saturated core: you want the code that is actually executing, without the waits burying it. The classic mistake is answering a latency question with a CPU profile, seeing a flat, cheap-looking graph, and concluding there is no problem.

code

python · 13 lines
python
import time


def measure(label, fn):
    wall0, cpu0 = time.perf_counter(), time.thread_time()
    fn()
    wall = time.perf_counter() - wall0
    cpu = time.thread_time() - cpu0
    print(f"{label:<10} wall={wall:.2f}s cpu={cpu:.2f}s")


measure("blocked", lambda: time.sleep(0.5))
measure("computing", lambda: sum(i * i for i in range(3_000_000)))

go deeper

for a junior

Remember the one-line difference: wall-clock counts waiting, CPU time does not. If the code is sitting on a network call, only a wall-clock profile will show it.

for a middle

Explain the trigger behind each: an elapsed-time interval versus a timer charged against CPU consumed, and be able to pick the right one from a symptom such as high latency with idle cores.

for a senior

Demonstrate that you check which clock a graph came from before acting, and can explain the CPython wrinkles — process-wide CPU timers, main-thread-only signal handlers, a sampler thread that needs the GIL.

for a principal

Decide which view the organisation's default profiling story ships with, since it silently shapes every conclusion drawn from it, and make sure latency and saturation investigations are not handed the same graph.

### Two different clocks Every sampling profiler needs something to tell it *when* to look. That choice, not the stack-walking code, is what makes a profile wall-clock or CPU-time. - **Wall-clock (elapsed time).** The trigger is real time: a sampler thread that waits on an interval, an interval timer tied to elapsed time, or an out-of-process sampler polling on its own schedule. Every thread it walks contributes a sample whatever it is doing — running, waiting on a socket, blocked on a lock, sleeping. - **CPU time.** The trigger is consumption: on Unix, `signal.setitimer(signal.ITIMER_PROF, ...)` arms a timer that advances only while the process is on a CPU, delivering `signal.SIGPROF` when it expires. A thread parked in a blocking read advances that timer not at all, so it produces no samples and vanishes from the profile. ### What each one answers A latency question — "this request takes 900 ms, where does it go?" — is a wall-clock question. Most of that 900 ms is usually not computation at all: it is a downstream call, a connection wait, a lock, a file read. Only a wall-clock profile shows it, and it shows it as a fat block of samples parked inside the blocking call. A throughput or saturation question — "we are pinned at 100% of a core, what is eating it?" — is a CPU-time question. Here waiting is not the story, and wall-clock samples of idle threads dilute the signal you want. Sampling only when the process is actually running keeps the profile focused on executing code. ### Doing the same distinction in code The same split exists in Python's timing functions, which is the cheapest way to build intuition: `time.perf_counter()` measures elapsed wall-clock time; `time.process_time()` measures CPU time for the whole process; `time.thread_time()` measures CPU time for the calling thread. Wrap `time.sleep(0.5)` in both and the wall figure is 0.5 s while the CPU figure is near zero. Wrap a tight arithmetic loop and the two nearly agree. A profile is that same measurement, taken continuously and attributed to stacks. ### Threads, the GIL and signal delivery Two CPython details complicate CPU-time sampling. First, `signal.ITIMER_PROF` is charged per *process*, not per thread, so in a multi-threaded worker the timer advances whenever any thread is running, and attributing the resulting sample to the right thread takes care. Second, CPython runs Python-level signal handlers only on the main thread, at a bytecode boundary, once the GIL is available — so if the main thread sits inside a long C call, handler execution is deferred and samples bunch up after the fact. An in-process wall-clock sampler running on its own thread has a related failure: it also needs the GIL, so a C extension that holds the GIL throughout a long computation stalls it. A C extension that releases the GIL around that computation is sampled cleanly, and time inside it appears against the Python frame that called it. ### Where a resource leak shows up The two views also disagree in a diagnostically useful way. Suppose an image-thumbnail worker at a 1,200-request-per-minute peak leaks file handles because a resource is left unclosed on an error path. Under a CPU-time profile there is nothing to see: the worker is not burning cycles, it is failing and retrying. Under a wall-clock profile the same run shows a growing share of samples parked in the code that opens or closes those resources, and in the garbage collector or finalizer path that eventually reclaims them. Same process, same sampler, completely different conclusion — because the two clocks disagree about whether waiting counts. ### Rules of thumb Start wall-clock. It is the default of most out-of-process samplers, it answers the question users actually ask, and it cannot hide a problem by declining to sample it. Switch to CPU time when a wall-clock profile is dominated by waits you have already explained and you need to see the compute underneath, or when you are optimising a batch job that is genuinely CPU-bound. Always know which one you are looking at before you draw a conclusion: the graphs look identical and mean opposite things. ### In an interview Say what triggers each sampler, give the one-line consequence for blocked threads, and pick the right one for a stated symptom. If you add the CPython wrinkle that Python signal handlers only run on the main thread, you are clearly speaking from having built or debugged one.

  • A service is slow but every core is nearly idle. Which profile do you take, and why?
    A wall-clock profile. Idle cores mean the time is going into waiting — downstream calls, locks, disk, connection pools — and a CPU-time sampler will not fire while a thread is blocked, so it would produce a small, clean, useless graph. Wall-clock sampling parks samples inside the blocking call and names the wait directly.
  • Why can a CPU-time sampler built on signal.setitimer miss what a worker thread is doing?
    Two reasons. `signal.ITIMER_PROF` is charged to the whole process rather than to one thread, so the timer advances on any thread's CPU use. And CPython executes Python-level signal handlers only on the main thread, at a bytecode boundary — so the stack that gets captured is the main thread's, not the busy worker's, and delivery is deferred while the main thread is inside a long C call.
  • Which Python calls let you check the wall-clock/CPU split for a block of code without a profiler?
    `time.perf_counter()` for elapsed time, `time.process_time()` for whole-process CPU time, and `time.thread_time()` for the calling thread's CPU time. Taking a pair before and after a suspect block tells you immediately whether it is waiting or computing, which usually decides which kind of profile to take next.

A wall-clock profile is a stopwatch on the whole errand, including the queue at the counter; a CPU-time profile is a stopwatch that runs only while you are actually being served.

saying these in an interview costs you the question

  • Believes both profiles show the same total time
  • Diagnoses a latency problem with a CPU-time profile
  • Thinks blocked threads still consume CPU time
  • Cannot say what triggers each sampler
  • Assumes signal-driven sampling sees every thread

context

resolving links…