urllib and http.client

## Two failures that feel the same and are not When `urllib.request.urlopen` does not give you a response, exactly one of two things happened. Either the conversation never got far enough for the server to say anything — DNS did not resolve, the TCP connection was refused or reset, the TLS handshake failed — or the server answered perfectly well and the answer was a status urllib treats as an error. urllib gives these different exception types, and the whole point of the distinction is that they call for different reactions: the first is a transport problem you might retry or route around, the second is an answer you must read. ## The hierarchy ``` OSError └─ urllib.error.URLError └─ urllib.error.HTTPError (also a readable response object) ``` `URLError` subclasses `OSError`, so any handler that already catches `OSError` catches it. `HTTPError` subclasses `URLError`. That single inheritance edge is the source of the most-asked question on this topic: because a `HTTPError` *is a* `URLError`, writing the general clause first silently absorbs the specific one. ```python try: resp = urllib.request.urlopen(url, timeout=5) except urllib.error.HTTPError as exc: # must come first ... except urllib.error.URLError as exc: # transport failure ... ``` Reverse those two clauses and you can no longer tell a 503 from a refused connection, because both land in the same branch and only one of them has a status code. ## HTTPError is also the response `HTTPError` is unusual: besides subclassing `URLError`, it inherits from urllib's response wrapper, which makes the exception object a file-like response. Inside the handler you can use the `code` attribute for the numeric status, `reason` for the status text, `headers` for the response headers, and `read()` for the body bytes. That last one matters more than it sounds: an API that explains *why* it rejected you does so in the body of the 4xx, and the only place that body exists is on the exception instance. Read it inside the `except` block — once the object is discarded, the socket goes with it. The `reason` attribute means different things on the two classes. On `HTTPError` it is the status phrase the server sent. On `URLError` it is usually the underlying exception object — a `socket.gaierror` from a failed lookup, a `ConnectionRefusedError`, an SSL error — so log its `repr` or check its type rather than matching on message text, which varies by platform and resolver. ## Which statuses raise at all urllib raises `HTTPError` for statuses outside the 2xx range, with one large exception: redirect statuses are consumed by the redirect handler and followed rather than raised, so a followed redirect looks to you like the final 200. If the redirect chain hits urllib's limit, or the handler declines to follow it, then it surfaces as an `HTTPError` with the 3xx code. This is why there is no `raise_for_status()` idiom in the standard library. The raising happens automatically, and the interesting work is deciding, in the handler, which codes you treat as fatal, which as retryable, and which as an expected answer that happens to be non-2xx. ## The timeout hole Here is the part that separates someone who has read the docs from someone who has debugged this. urllib wraps the errors raised while it connects and sends the request into a `URLError`. It does not wrap what happens next. A timeout that fires while waiting for the response status line propagates out of `urlopen` as a plain `TimeoutError`, and `TimeoutError` and `URLError` are siblings under `OSError` — neither one catches the other. Code that carefully handles `HTTPError` and `URLError` and nothing else will crash on the single most common production failure mode. Two defensible fixes: catch `OSError` as the outermost transport clause (it covers `URLError`, `TimeoutError` and anything else the socket layer throws), or name `TimeoutError` explicitly alongside `URLError` so the retry path is obvious to a reader. Since Python 3.10, `socket.timeout` is an alias of the builtin `TimeoutError`, so there is only one name to catch. ## What to say in an interview State the inheritance chain, state that `HTTPError` is simultaneously an exception and a response, state the clause ordering rule, then show judgement: a 5xx and a refused connection may both be retryable, but only one of them tells you the service is alive; a 4xx is almost never worth retrying and its body is the diagnostic. Mentioning the bare `TimeoutError` on the read path is the detail that marks real operational experience.