A round trip through url.Parse and url.Values.Encode broke our signed callback URLs — what does it not preserve?

seniornice to knowfreq 30%

answer

the signature covers bytes, not meaning
a map cannot remember an order
re-encoding sorts and re-escapes
percent twenty in, plus out
verify before anything normalises

basics

~20 s

Parsing a query into url.Values and re-encoding it canonicalises the bytes: parameter order becomes sorted by key, a space sent as %20 comes back as +, unnecessary escapes are dropped, and a valueless key gains a trailing equals sign. A signature over the original bytes then fails.

solid answer

~50 s

url.Parse alone is faithful — u.String() writes RawQuery out verbatim. The damage comes from u.Query() followed by v.Encode(). Query() decodes into a map[string][]string, which cannot hold the original ordering, and it silently drops pairs it cannot parse. Encode() then re-emits with its own rules: keys sorted, every value escaped by url.QueryEscape, so a space that arrived as %20 leaves as +, an unnecessarily escaped %7E leaves as ~, and ?flag becomes flag=. Every one of those is the same URL semantically and a different byte string, which is all a signature cares about. The fix is to verify against the bytes as received — r.URL.RawQuery before any normalisation — or, if you must canonicalise, define the canonical form once and have both the signer and the verifier produce it, so the transformation is idempotent. A fuzz target that parses a URL, re-encodes it and compares the two strings finds the remaining cases.

`url.Parse` alone is faithful — `u.String()` writes `RawQuery` out verbatim. The damage comes from `u.Query()` followed by `v.Encode()`. `Query()` decodes into a `map[string][]string`, which cannot hold the original ordering, and it silently drops pairs it cannot parse. `Encode()` then re-emits with its own rules: keys sorted, every value escaped by `url.QueryEscape`, so a space that arrived as `%20` leaves as `+`, an unnecessarily escaped `%7E` leaves as `~`, and `?flag` becomes `flag=`. Every one of those is the same URL semantically and a different byte string, which is all a signature cares about. The fix is to verify against the bytes as received — `r.URL.RawQuery` before any normalisation — or, if you must canonicalise, define the canonical form once and have both the signer and the verifier produce it, so the transformation is idempotent. A fuzz target that parses a URL, re-encodes it and compares the two strings finds the remaining cases.

code

go · 7 lines

raw := "https://api.example.test/cb?b=2&a=h%20i&flag"
u, _ := url.Parse(raw)
fmt.Println(u.String() == raw)
// true - String writes RawQuery verbatim
u.RawQuery = u.Query().Encode()
fmt.Println(u.RawQuery)
// a=h+i&b=2&flag=

go deeper

for a junior

Recall that url.Values.Encode sorts keys and escapes values with its own rules, so the string it produces need not match the string that was parsed.

for a middle

Explain each transformation: sorted keys, a space re-encoded as a plus, unnecessary escapes removed, a valueless key gaining an equals sign, malformed pairs dropped. Know that String itself writes RawQuery verbatim.

for a senior

Diagnose it end to end: compare the received query with the bytes the verifier hashed, find the rewriting step, and move verification ahead of any normalisation. Reach for a fuzz target that asserts the round trip is the identity.

for a principal

Own the scheme rather than the bug. Decide whether URLs are signed as raw bytes with a no-rewrite rule, or over a canonical form both sides compute, and write down which parameters participate so a later addition cannot invalidate every link in flight.

Where the bytes change, and where they do not

It helps to separate two operations that are easy to conflate.

Parsing and re-rendering is faithful for the query. u.String() writes the query as ? followed by RawQuery, verbatim, with no re-escaping. The path goes through EscapedPath(), which returns the stored RawPath when it is still a valid escaping of Path. So url.Parse followed immediately by String() gives you back what you had.

Going through url.Values is not faithful. u.Query() parses RawQuery into a map[string][]string, and v.Encode() renders that map. The map is the lossy step, and Encode is opinionated about the output. Together they normalise:

Order. A map has no order, so Encode sorts by key. b=2&a=1 becomes a=1&b=2. Values under a single key keep their relative order, but the interleaving between different keys is gone.
Space encoding. Encode escapes each value with url.QueryEscape, which writes a space as +. A sender who wrote %20 gets + back.
Escaping strictness. Decoding then re-encoding removes escapes that were not required (%7E decodes to ~, and ~ is unreserved so it is emitted bare) and adds escapes for characters the sender left literal.
Valueless keys. ?flag parses to one key with an empty value and re-encodes as flag=.
Malformed pairs. Query() silently discards pairs it cannot parse, so they vanish from the output entirely rather than raising an error.

Every one of those transformations produces a URL that means the same thing. A signature does not check meaning; it checks bytes. So a refactor that innocently introduces a parse-and-rebuild step in the middle of the request path — a helper that adds a tracing parameter, a middleware that normalises the URL, a client wrapper that rebuilds the request — invalidates every signature computed over the original string, without changing a single parameter value.

Diagnosing it

The symptom is confusing because nothing looks wrong: the parameters are all present and all correct, and the key has not rotated. Print the two strings next to each other — the received query and the one the verifier actually hashed — and the diff is usually a single character. %20 against + and a reordering are the two you will see most.

The systematic version is a fuzz target. Parse the input, re-encode the query through Values, and assert that the rendered URL is unchanged; anything that fails is a case where your canonicalisation is not the identity:

func FuzzQueryRoundTrip(f *testing.F) {
	f.Add("https://api.example.test/cb?b=2&a=h%20i")
	f.Fuzz(func(t *testing.T, raw string) {
		u, err := url.Parse(raw)
		if err != nil {
			t.Skip()
		}
		got := *u
		got.RawQuery = u.Query().Encode()
		if got.String() != u.String() {
			t.Errorf("round trip changed %q to %q", u.String(), got.String())
		}
	})
}

Run it with go test -fuzz=FuzzQueryRoundTrip. It will produce failures immediately, which is the point: it makes the set of transformations concrete instead of a list somebody has to remember.

The two ways to make it stop

Verify over the received bytes. The signature covers a string; keep that string. On the receiving side that means reading r.URL.RawQuery (or the raw request target) before anything normalises it, and computing the MAC over exactly those bytes. Nothing in the chain is then allowed to rewrite the URL before verification — which is a constraint worth writing down, because the next refactor will try.

Or canonicalise deliberately, on both sides. If the URL must survive proxies and rewrites that you do not control, do not sign the URL as text. Define a canonical string — sorted keys, one chosen escaping, a fixed subset of parameters — and have the signer build it and the verifier rebuild it with the same code. The property you need is idempotence: canonicalising twice must equal canonicalising once. Values.Encode is a reasonable base for that, because sorting and re-escaping are idempotent, provided you also fix which parameters participate so that a later addition (a tracing parameter, a cache-buster) does not change the input.

A third habit helps either design: put the signature parameter itself outside the signed set, and decide explicitly what happens to unknown parameters, rather than letting whichever code runs first decide for you.

The general lesson

url.Values is a model of a query, not a copy of one. Modelling is exactly what you want when you are building a URL, and exactly what you do not want when the bytes themselves carry meaning. Whenever a URL is an input to a hash, a cache key, an authorisation decision or a log-based comparison, ask which of those two you are doing before you call Query().

## Where the bytes change, and where they do not It helps to separate two operations that are easy to conflate. **Parsing and re-rendering is faithful for the query.** `u.String()` writes the query as `?` followed by `RawQuery`, verbatim, with no re-escaping. The path goes through `EscapedPath()`, which returns the stored `RawPath` when it is still a valid escaping of `Path`. So `url.Parse` followed immediately by `String()` gives you back what you had. **Going through `url.Values` is not faithful.** `u.Query()` parses `RawQuery` into a `map[string][]string`, and `v.Encode()` renders that map. The map is the lossy step, and `Encode` is opinionated about the output. Together they normalise: - **Order.** A map has no order, so `Encode` sorts by key. `b=2&a=1` becomes `a=1&b=2`. Values under a single key keep their relative order, but the interleaving between different keys is gone. - **Space encoding.** `Encode` escapes each value with `url.QueryEscape`, which writes a space as `+`. A sender who wrote `%20` gets `+` back. - **Escaping strictness.** Decoding then re-encoding removes escapes that were not required (`%7E` decodes to `~`, and `~` is unreserved so it is emitted bare) and adds escapes for characters the sender left literal. - **Valueless keys.** `?flag` parses to one key with an empty value and re-encodes as `flag=`. - **Malformed pairs.** `Query()` silently discards pairs it cannot parse, so they vanish from the output entirely rather than raising an error. Every one of those transformations produces a URL that means the same thing. A signature does not check meaning; it checks bytes. So a refactor that innocently introduces a parse-and-rebuild step in the middle of the request path — a helper that adds a tracing parameter, a middleware that normalises the URL, a client wrapper that rebuilds the request — invalidates every signature computed over the original string, without changing a single parameter value. ## Diagnosing it The symptom is confusing because nothing looks wrong: the parameters are all present and all correct, and the key has not rotated. Print the two strings next to each other — the received query and the one the verifier actually hashed — and the diff is usually a single character. `%20` against `+` and a reordering are the two you will see most. The systematic version is a fuzz target. Parse the input, re-encode the query through `Values`, and assert that the rendered URL is unchanged; anything that fails is a case where your canonicalisation is not the identity: ```go func FuzzQueryRoundTrip(f *testing.F) { f.Add("https://api.example.test/cb?b=2&a=h%20i") f.Fuzz(func(t *testing.T, raw string) { u, err := url.Parse(raw) if err != nil { t.Skip() } got := *u got.RawQuery = u.Query().Encode() if got.String() != u.String() { t.Errorf("round trip changed %q to %q", u.String(), got.String()) } }) } ``` Run it with `go test -fuzz=FuzzQueryRoundTrip`. It will produce failures immediately, which is the point: it makes the set of transformations concrete instead of a list somebody has to remember. ## The two ways to make it stop **Verify over the received bytes.** The signature covers a string; keep that string. On the receiving side that means reading `r.URL.RawQuery` (or the raw request target) before anything normalises it, and computing the MAC over exactly those bytes. Nothing in the chain is then allowed to rewrite the URL before verification — which is a constraint worth writing down, because the next refactor will try. **Or canonicalise deliberately, on both sides.** If the URL must survive proxies and rewrites that you do not control, do not sign the URL as text. Define a canonical string — sorted keys, one chosen escaping, a fixed subset of parameters — and have the signer build it and the verifier rebuild it with the same code. The property you need is idempotence: canonicalising twice must equal canonicalising once. `Values.Encode` is a reasonable base for that, because sorting and re-escaping are idempotent, provided you also fix which parameters participate so that a later addition (a tracing parameter, a cache-buster) does not change the input. A third habit helps either design: put the signature parameter itself outside the signed set, and decide explicitly what happens to unknown parameters, rather than letting whichever code runs first decide for you. ## The general lesson `url.Values` is a *model* of a query, not a copy of one. Modelling is exactly what you want when you are building a URL, and exactly what you do not want when the bytes themselves carry meaning. Whenever a URL is an input to a hash, a cache key, an authorisation decision or a log-based comparison, ask which of those two you are doing before you call `Query()`.

Does url.Parse followed by u.String() change the query at all?
No. `String()` writes `RawQuery` verbatim, without re-escaping, and renders the path through `EscapedPath()`, which preserves the original escaping when it is still valid. The change only appears once you go through `u.Query()` and `Values.Encode()`, because the map cannot carry ordering and `Encode` applies its own escaping rules.
What happens to a parameter written as ?flag with no value?
`u.Query()` records it as the key `flag` with a single empty value, and `Encode()` writes it back as `flag=`. Semantically equivalent, one byte different. `Values.Has` is how you distinguish a present-but-empty key from an absent one, since `Get` returns an empty string for both.
If the URL must survive proxies you do not control, what do you sign instead?
Not the URL as text. Define a canonical string — a fixed set of parameters, sorted, with one chosen escaping — and have signer and verifier build it with the same code, so canonicalising twice equals canonicalising once. Keep the signature parameter itself out of the signed set, and decide explicitly whether unknown parameters are ignored or rejected.
Why is a fuzz target a better diagnostic here than a table of examples?
The transformations are a long tail: ordering, `+` against `%20`, unnecessary escapes, valueless keys, malformed pairs that get dropped. A target that parses, re-encodes and compares asserts the property you actually want — that your normalisation is the identity — and finds the cases nobody thought to put in the table.

Re-encoding a URL is like retyping a signed letter in your own handwriting. Every word is identical, but the signature was over the ink.

saying these in an interview costs you the question

Assumes parse then re-encode is byte-identical
Thinks Values.Encode preserves the original parameter order
Signs a re-serialised URL rather than the received bytes
Believes %20 and + are interchangeable everywhere
Blames the key or the clock when the canonical form changed
Does not know u.Query() silently drops malformed pairs

A round trip through url.Parse and url.Values.Encode broke our signed callback URLs — what does it not preserve?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Where the bytes change, and where they do not

Diagnosing it

The two ways to make it stop

The general lesson

▾common follow-ups4 questions

▾analogy~10 s

▾red flags6 flags

Where the bytes change, and where they do not

Diagnosing it

The two ways to make it stop

The general lesson

answer

go deeper