A metrics scraper passes timeout=2 to urllib.request.urlopen yet a call still runs for minutes. What does that timeout actually bound?
answer
- Not a stopwatch on the request
- Per operation, not per call
- The clock restarts on every byte
- No argument means no timeout at all
- A deadline must live outside urlopen
basics
~20 sIt is a socket timeout applied to each blocking operation — the connect and every individual read — not a deadline for the whole exchange. A target trickling bytes resets it indefinitely, so a total budget has to be enforced outside the call.
solid answer
~50 sThe `timeout` argument is handed to the underlying connection and becomes the socket timeout, so it fires only when one socket operation makes no progress for that long. The connect, the wait for the status line and every `read()` each get a fresh two seconds; a target dripping one byte per second never trips it, and total call time is unbounded. Omit the argument and you inherit the socket module's own default, which is `None` — block forever — unless something called `socket.setdefaulttimeout`. For a scraper with a per-target latency budget you need a real deadline: read the body in chunks against a `time.monotonic()` deadline and abandon it, cap the bytes you will accept, and run the fetch somewhere you can walk away from. Note also that a stall waiting for the response raises `TimeoutError`, not `URLError`.
code
python · 24 linesimport socket
import threading
import time
import urllib.request
listener = socket.create_server(("127.0.0.1", 0))
def drip_one_byte_at_a_time():
conn, _ = listener.accept()
conn.recv(4096)
conn.sendall(b"HTTP/1.0 200 OK\r\nContent-Length: 10\r\n\r\n")
for _ in range(10):
conn.sendall(b"x")
time.sleep(0.3)
conn.close()
threading.Thread(target=drip_one_byte_at_a_time, daemon=True).start()
start = time.monotonic()
with urllib.request.urlopen(f"http://127.0.0.1:{listener.getsockname()[1]}/", timeout=0.5) as resp:
body = resp.read()
print(len(body), "bytes in", round(time.monotonic() - start, 1), "s, no timeout raised")
listener.close()go deeper
Always pass timeout explicitly to urlopen. Without it the call can block indefinitely, which is the most common reason a small script appears to hang with no error at all.
Explain that the value becomes a socket timeout applied to each blocking operation, and that omitting it inherits the socket module's default of no timeout rather than some safe built-in number.
Show how you put a wall-clock budget above the socket timeout, why a drip-feeding target defeats per-read timeouts, and how you make an abandoned fetch visible as a failure instead of a gap in the data.
Own the timeout policy across the fleet: budgets derived from the latency objective rather than folklore, what an abandoned fetch does to downstream series, and whether one pathological target may consume a shared worker pool.
## What the argument really sets `urllib.request.urlopen(url, timeout=2)` does not start a stopwatch on the request. The value is passed down to the connection object and installed as the socket's timeout, which is a **per-operation** limit: a blocking socket call raises `TimeoutError` if it makes no progress for that many seconds. Every operation gets its own fresh window. Connecting gets two seconds. Waiting for the status line gets two seconds. Each read of the body gets two seconds, measured from the last byte that arrived. The consequence is arithmetic, not opinion: total time is bounded by (number of operations) x timeout, and the number of reads is controlled by the peer. A server that sends one byte every second, forever, keeps a `timeout=2` call alive forever while never once being idle for two seconds. ## The scenario A metrics scraper polls a few hundred targets on a fixed interval, with a per-target budget derived from a 92nd-percentile latency objective. Every worker passes `timeout=2`, and the team calls that the budget. Then one target starts responding slowly — a stuck exporter emitting its payload a chunk at a time. The scrape does not fail; it simply takes four minutes, holding a worker the whole time. The series for that target does not show an error, it shows a hole, and the samples that resume afterwards carry timestamps that make the gap look like a clock-skew artefact on the collector rather than a stuck fetch. The scraper's own latency histogram is quietly wrong too, because a scrape that is still running has not contributed a sample at all: the 92nd percentile only sees the calls that finished. The diagnosis chain is: a socket timeout is not a deadline; the budget was never enforced; and because nothing recorded the abandonment, the failure disguised itself as a data problem. ## The no-argument case is worse Omit `timeout` and `urlopen` uses a sentinel meaning *whatever the socket module's default is*. That default is `None` unless the process called `socket.setdefaulttimeout`, and `None` means block indefinitely. A single hung peer then pins the calling thread until the kernel or the peer eventually gives up, which can be many minutes. In a fixed worker pool, a handful of such peers takes the whole pool down while every health check stays green. A process-wide `socket.setdefaulttimeout` call is a blunt backstop — it changes the default for every socket in the process, including libraries you did not write — but as a floor against 'no timeout at all' it is defensible in a small scraper, and it is worth naming in an interview as a deliberate, coarse safety net rather than the primary control. ## Bounding the body `urlopen` returning is not the end of the call. The response arrives lazily; `read()` is where most of the wall clock usually goes, and the socket timeout still applies per read. Two guards belong there: * a wall-clock deadline. Take `time.monotonic()` before the call, read the body in chunks, and stop when the deadline passes. What you do then is a policy choice — discard the sample, record a failure — but you must stop. * a byte cap. A response with no length, or a lying length, can otherwise fill memory. Refuse to accumulate more than a fixed number of bytes. An abandoned read leaves the connection to be closed, which is fine; urllib opens a fresh connection per call anyway, sending `Connection: close`. ## Enforcing a hard deadline If the budget must be honoured even when the peer is pathological, the fetch has to run somewhere you can leave behind. Submit it to a worker and stop waiting for the result once the budget expires, treating the abandoned scrape as a failed sample. A thread cannot be killed, so the abandoned worker keeps running until its socket timeout finally fires — which is exactly why the socket timeout stays in place as the inner guard, and why the pool must be sized for the possibility. Where the guarantee has to be absolute, the fetch goes in a separate process you can terminate. ## The exception it raises One more detail that decides whether error handling actually works. urllib wraps errors from the connect and send phases into `urllib.error.URLError`, but a timeout that fires while waiting for the response status line comes out of `urlopen` as a bare `TimeoutError`. Both are `OSError` subclasses and neither is a subclass of the other, so an `except urllib.error.URLError` clause does not catch the timeout. Since 3.10, `socket.timeout` is an alias of the builtin `TimeoutError`, so `except (urllib.error.URLError, TimeoutError)` — or simply `except OSError` — is the shape that survives contact with a slow target. ## The summary an interviewer wants The timeout argument is a liveness guard on individual socket operations. Latency budgets are wall-clock, so they need a deadline you enforce yourself, a byte cap, and an explicit record when a fetch is abandoned — because an unrecorded abandonment shows up downstream as a mysterious gap rather than as the failure it is.
- What timeout applies when urlopen is called without the timeout argument?Its default is a sentinel meaning 'use the socket module's own default', and that default is `None` unless the process called `socket.setdefaulttimeout`. `None` means block indefinitely, so one hung peer pins the calling thread until the kernel or the peer gives up. On a fixed worker pool that is how every worker disappears while the process still looks healthy.
- Does the timeout cover the time spent inside the read of the response body?Only per read. The socket timeout stays on the connection, so each read raises `TimeoutError` when no bytes arrive within the window, but a body that keeps arriving slowly can take arbitrarily long in total. Bound it yourself: read in chunks against a `time.monotonic()` deadline, and cap the total bytes you will accept so a response with no honest length cannot exhaust memory.
- How do you enforce a hard per-target deadline across a pool of scrape workers?Run the fetch somewhere you can walk away from — a worker whose result you stop waiting for once the budget expires, or a separate process you can terminate — and count the abandoned scrape as a failed sample rather than blocking the scheduler. Keep the socket timeout as the inner guard so the orphaned connection eventually dies, and record the abandonment explicitly so the gap in the series reads as a failure instead of a clock problem.
A socket timeout is a patience limit on each sip, not a limit on how long lunch may take; a companion who sips steadily can keep you at the table all afternoon.
saying these in an interview costs you the question
- Calling the timeout a deadline for the whole request
- Assuming urlopen has a sensible default timeout
- Thinking a slow trickle of bytes trips a socket timeout
- Expecting a stalled read to raise URLError
- Bounding connect time while ignoring the body read
- Reading an unbounded response body into memory