Using urllib.request, how do you send a POST with custom headers and a body?

## Two objects, one call `urllib.request` splits an HTTP call into a **request object** and a **call that performs it**. `urlopen` is the performer; it takes either a plain URL string or a `urllib.request.Request` instance. Everything about the request other than the URL, the body and the timeout lives on the `Request`: the headers, the method override, the proxy flags. This is the first thing to internalise, because the most common beginner mistake is reaching for a `headers` keyword on `urlopen` that does not exist and never has. ```python import json import urllib.request payload = json.dumps({"scrape": "ok"}).encode("utf-8") req = urllib.request.Request( "https://example.invalid/ingest", data=payload, headers={"Content-Type": "application/json"}, ) # urlopen(req, timeout=5) would send it ``` ## The body decides the verb `Request` takes `data=None` by default. When `data` is not `None`, urllib's HTTP handler treats the request as a POST: the method returned by the request object is `POST` rather than `GET`. There is no separate switch. If you want a different verb you pass `method="PUT"` (or `"DELETE"`, `"PATCH"`, `"HEAD"`) to the constructor, and that explicit method wins over the data-implies-POST rule. `urlopen` itself has no method parameter at all, so a verb other than GET always means you built a `Request`. ## The body must be bytes HTTP bodies are octets, and urllib refuses to guess an encoding for you. Passing a `str` raises `TypeError` with the message that POST data should be bytes, an iterable of bytes, or a file object. Note *when* it is raised: the constructor happily stores the string, and the error surfaces later, when the handler prepares the request — so the traceback points at `urlopen`, not at the line where you built the `Request`. The two everyday encodings are: * form-encoded — `urllib.parse.urlencode({"k": "v"}).encode("utf-8")`, producing `b"k=v"`; * JSON — `json.dumps(obj).encode("utf-8")`, and you must set `Content-Type` yourself, because urllib will not infer it from the payload. A file object or any iterable of bytes is also accepted. When urllib cannot determine a length for the body it switches to chunked transfer encoding instead of sending `Content-Length`. ## What urllib adds for you Just before the request goes out, the HTTP handler tops up the headers. If you supplied a body and no content type, it adds `Content-Type: application/x-www-form-urlencoded` — which is right for form data and wrong for everything else, so set it explicitly for JSON. It adds `Content-Length` when the body has a length, `Host` derived from the URL, `Connection: close` (urllib does not keep connections alive; every call is a fresh connection), and a default `User-Agent` of `Python-urllib/` plus the interpreter's major and minor version. Many public endpoints reject that user agent, which is why a request that works from a browser can fail from a script until you override the header. These fill-ins are attached as *unredirected* headers, meaning they are deliberately not copied onto a follow-up request if the response is a redirect. ## The header-casing surprise Headers you pass in the `headers` dict all go through `add_header`, which stores each key as `key.capitalize()`. `X-Scrape-Id` becomes `X-scrape-id`. On the wire this is harmless, because HTTP header names are case-insensitive, but it matters twice: when you inspect the request object in a test, and when you call `has_header`, `get_header` or `remove_header`, all of which look up the capitalized spelling. ## Reading the response For `http` and `https` URLs, `urlopen` returns an `http.client.HTTPResponse`. It is a context manager and a binary file-like object, so the idiomatic shape is: ```python import urllib.request with urllib.request.urlopen("https://example.invalid/", timeout=5) as resp: status = resp.status body = resp.read().decode("utf-8") ``` `read()` returns `bytes`; decoding is your job, using the charset from the response headers if you care about correctness. The `status` attribute holds the numeric status, `headers` the response headers, and `url` the URL that was actually fetched. ## What the standard library will not do The stdlib client is deliberately thin. There is no connection pooling, no retry policy, no session object holding cookies by default, no automatic JSON encoding or decoding, and no timeout unless you pass one. That is the honest trade: zero dependencies, and every policy decision left to you. Knowing where the line falls is what an interviewer is checking for — the mechanics above are enough for a health check, a metrics scrape or a build script, and reaching for something heavier is a judgement call rather than a reflex.