skip to content

Assets served from an embed.FS come back stale after a deploy — why, and how do you fix it?

level: seniorimportance: nice to knowfreq 35%

answer

  1. the bytes changed, the headers did not
  2. compiled-in files have no timestamp
  3. one header is omitted, the other is never invented
  4. clients guess a lifetime when told nothing
  5. make the URL change with the bytes

basics

~20 s

Embedded files report a zero modification time, so http.ServeContent sends no Last-Modified header, and net/http never invents an ETag. With no validator and no cache directives, browsers apply heuristic freshness and keep the old copy. Fix it with content-hashed filenames or a build-derived ETag.

solid answer

~60 s

`http.FileServer` and `http.ServeFile` both end up in `http.ServeContent`, which takes its `Last-Modified` header from the `FileInfo.ModTime()` of the file it is serving. An `embed.FS` reports the zero `time.Time`, and `ServeContent` deliberately omits the header and skips the modification-time check when the modtime is zero. `net/http` also never computes an `ETag` — it only honours one you set on `w.Header()` yourself. So an embedded asset goes out with no `ETag`, no `Last-Modified` and, unless you added one, no `Cache-Control`; a browser then falls back to heuristic freshness and reuses its copy for a while regardless of your deploy. Confirm it with `curl -sI` against the URL before and after a release and note the missing validator headers. The durable fix is to put a content hash in the asset filename and serve those with a long immutable `Cache-Control`, keeping the HTML that references them uncached. The cheap fix is middleware that sets an `ETag` derived from the build ID before delegating to the file server, so `If-None-Match` starts producing 304s.

code

go · 8 lines
go
// buildID is stamped in at link time and changes with every release.
func versioned(h http.Handler, buildID string) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("ETag", `"`+buildID+`"`)
		w.Header().Set("Cache-Control", "public, max-age=300")
		h.ServeHTTP(w, r) // ServeContent compares If-None-Match against it
	})
}

go deeper

for a junior

Know that a browser can keep serving an old copy of a file after you deploy, and that the server controls this with response headers. Recognise that a hard refresh fixing it points at the client's cache, not the server's files.

for a middle

Explain the mechanism: ServeContent takes Last-Modified from the file's ModTime and skips it when that is the zero time, and an embedded file's ModTime is zero. Add that net/http never generates an ETag on its own.

for a senior

Show the diagnosis and the fix: compare headers across a release with curl, rule out the proxy by hitting the origin, then choose between content-hashed filenames with immutable caching and a build-derived ETag, and justify the choice.

for a principal

Own the asset delivery policy across services: who fingerprints, what the immutable cache lifetime is, how a rollback interacts with long-lived caches, and whether the build pipeline change is worth the round-trips it removes.

## The symptom A documentation site ships as a single binary with its CSS, JavaScript and images compiled in via `//go:embed`. A release goes out. The new binary is definitely running — you can see the new version string — but a fraction of users keep rendering the site with the previous stylesheet, sometimes for hours. Hard-refreshing fixes it for the person who complains, which is exactly the shape of a client-side caching problem rather than a server-side one. ## Where the missing headers come from Follow the call chain. `http.FileServer` resolves the request path against its root and calls `http.ServeFile`-equivalent logic, which stats the entry and calls: ```go http.ServeContent(w, r, name, fi.ModTime(), content) ``` `ServeContent` has an explicit rule: **if `modtime` is the zero `time.Time` (or the Unix epoch), it does not send `Last-Modified` and does not evaluate the modification-time preconditions.** And an `embed.FS`'s `fs.FileInfo` reports precisely the zero `time.Time` — embedded content has no meaningful on-disk timestamp, and inventing one would be a lie that varied per build machine. The second half is that `net/http` **never generates an `ETag`**. `ServeContent` will compare `If-None-Match` against an `ETag` header, but only if your code put one in `w.Header()` before the call. Nothing in the standard library hashes the body for you. So the response carries `Content-Type`, `Content-Length`, `Accept-Ranges` — and no validator at all, and no `Cache-Control` unless you added one. ## Why an absent validator makes things *worse*, not neutral It is tempting to assume that with no caching headers a client always refetches. It does not. Faced with a response carrying neither a freshness directive nor a validator, a browser is permitted to apply *heuristic* freshness — inventing a lifetime of its own. And even when it decides to revalidate, it has nothing to revalidate *with*: no `If-None-Match` and no `If-Modified-Since` it can send, so a conditional request is impossible and every check would be a full download anyway. Sending nothing is not the same as sending "do not cache". ## Confirming it rather than guessing The cheapest confirmation is a header dump against the deployed URL, taken before and after a release: ``` curl -sI https://docs.example.com/static/app.js | grep -iE 'etag|last-modified|cache-control' ``` If that prints nothing on both sides, the diagnosis is settled — the server never gave the client anything to distinguish the two builds by. A second check confirms the mechanism: send an `If-Modified-Since` far in the future and watch a `200` with the full body come back rather than a `304`. A third, if you want to be thorough: `curl -r 0-99` still returns `206 Partial Content`, which proves the range machinery is alive and it really is only the validator that is missing. Note what this rules out: it is not a proxy, not the mux, not the embed patterns. The bytes in the binary are new. Only the client's decision to ask is wrong. ## The two fixes, in order of durability **Content-addressed filenames.** Build-time tooling renames `app.js` to `app.a91f3c2e.js` and rewrites the references in the HTML. The URL now changes whenever the bytes change, so the asset can be served with a very long `Cache-Control: public, max-age=31536000, immutable` and never revalidated at all, while the HTML that names it is served uncached. Stale assets become structurally impossible rather than merely unlikely, and you also delete the revalidation round-trips entirely. This is the answer if you are the person asked to make the site faster without changing its behaviour: it removes requests rather than making them cheaper. **A build-derived `ETag`.** If you cannot change the build pipeline, wrap the file server in middleware that sets `ETag` to a quoted build identifier before delegating. `ServeContent` then answers `If-None-Match` with `304 Not Modified` for unchanged builds, and the first request after a deploy misses and refetches. It costs one round-trip per asset per user per deploy, but it is correct, and it is about six lines. A per-file hash is better than a per-build ID if you can compute one — it means a release that changed only the JavaScript does not invalidate the images. You can compute those hashes at start-up by walking the embedded FS once, which is fast and happens exactly once per process. ## What not to do Do not reach for `time.Now()` as the modtime. It makes `Last-Modified` change on every response, defeats every cache including intermediaries, and looks like it works because the stale copies stop appearing. Do not set `Cache-Control: no-store` on assets globally either; it fixes staleness by deleting caching altogether, which is the opposite of the goal on a documentation site whose assets are most of its bytes.

  • Why not just pass time.Now() as the modtime so Last-Modified is always fresh?
    It changes on every single response, so no client or intermediary can ever get a 304 and every request is a full download. Staleness disappears because caching disappears. It is a validator that can never validate anything.
  • How would you distinguish this from a reverse proxy or CDN holding the old copy?
    Request the origin directly, bypassing the edge, and compare the bytes and headers with what the edge returns. If the origin already serves the new bytes with no ETag, no Last-Modified and no Cache-Control, the caching decision is being made downstream because you gave it nothing to work with.
  • Why is a per-file hash better than one build-wide ETag?
    A build-wide identifier invalidates every asset on every release, so a change to one stylesheet forces every image and script to be refetched. Per-file hashes keep untouched assets valid across deploys, which is most of the bytes on a documentation site.
  • Where should the ETag be set relative to the file server?
    In middleware that sets it on w.Header() before calling the file server's ServeHTTP. ServeContent reads the ETag out of the response header when it evaluates If-None-Match, so setting it afterwards would be far too late — the response has already been written.

saying these in an interview costs you the question

  • Assumes a response with no cache headers is never cached
  • Thinks net/http computes an ETag from the file contents
  • Passes time.Now() as the modtime to force freshness
  • Blames the proxy without checking the origin's headers
  • Believes embedded files inherit their on-disk timestamps
  • Fixes it by disabling caching for all assets