skip to content

A Go service's embed.FS is missing files that exist in the source tree. How do you diagnose it?

level: seniorimportance: should knowfreq 40%

answer

  1. the contents were decided on the build machine
  2. a star does not cross a slash
  3. some names are skipped by rule
  4. list the filesystem, not the directory
  5. the all prefix turns the skipping off

basics

~20 s

Walk the embedded filesystem and diff it against the source tree; the file set was fixed at build time. The usual causes are a star glob, which never crosses a slash, and dot- or underscore-prefixed names, skipped unless the pattern says all:.

solid answer

~50 s

Start from the fact that the contents were decided on the build machine, so nothing you change at run time can help — the question is which files the pattern selected. I dump the filesystem at boot with `fs.WalkDir(assets, ".", ...)` and diff that against a listing of the directory in the repository. Two rules account for most gaps. A pattern like `web/*.html` uses `path.Match`, whose `*` does not cross a slash, so files in subdirectories were never candidates; naming the directory instead embeds the subtree recursively. And embedding a directory skips every entry whose name begins with `.` or `_`, at any depth, unless the pattern is written `all:web`. After that I check whether the file exists in a clean checkout at all — an ignored or generated asset that is present on a laptop makes the build fail or come up short in the pipeline.

code

go · 12 lines
go
//go:embed all:web
var assets embed.FS

func dumpEmbedded() error {
	return fs.WalkDir(assets, ".", func(p string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}
		fmt.Println(p, d.IsDir())
		return nil
	})
}

go deeper

for a junior

Recall the two rules that hide files: a star in a pattern does not reach into subdirectories, and embedding a directory skips names starting with a dot or an underscore.

for a middle

Be able to explain that pattern resolution is a build-time step, so the fix is a rebuild with a corrected pattern, and to say what the all: prefix changes.

for a senior

Drive the investigation from evidence: walk the embedded filesystem and diff it against the source tree, distinguish an empty filesystem from a partial one, and check whether a clean checkout even contains the asset.

for a principal

Turn the incident into a rule: prefer directory patterns over file globs, and require a test that asserts the critical assets are present, so this failure class can only appear in the pipeline.

## The first move: separate build time from run time The contents of an `embed.FS` are fixed when the binary is built. There is no lookup path, no cache, no working directory involved at run time. That single fact eliminates most of the hypotheses people start with — the deployment did not copy something, the container image lost a layer, the working directory is wrong — and points the investigation at one question: **which files did the pattern actually select on the machine that built this binary?** ## The diagnostic: print the filesystem, not the directory The cheapest and most conclusive check is to walk the embedded filesystem and print every path, either behind a debug flag at startup or in a small test. `fs.WalkDir(assets, ".", fn)` visits the root and everything under it, giving the exact set of names the binary carries. Comparing that list with a listing of the source directory shows the gap immediately, and it distinguishes *missing* from *named differently*, which is the other common outcome — remember that names keep the pattern's directory prefix. Doing this in a test is better than doing it at startup, because it fails in the pipeline rather than in production. A test that asserts a handful of critical assets are present, or that the walk yields at least the expected count, catches the whole class of problem before release. ## Cause one: the glob does not descend Patterns use `path.Match` semantics, in which `*` matches a run of non-separator characters. It does not cross a `/`. So `//go:embed web/*.html` matches `web/index.html` and nothing in `web/partials/`. There is no `**` operator to reach for. The fix is to name the directory itself — `//go:embed web` — because when a pattern matches a directory the entire subtree beneath it is embedded recursively. A pattern like `web/*` also descends, since the entries it matches include directories, which then bring their subtrees. ## Cause two: the skipped names When a pattern matches a directory, the recursive walk **excludes every entry whose name begins with `.` or `_`**, at any depth. This is a deliberate default: it keeps version-control metadata, editor droppings, and Go's own convention for ignored directories out of the binary. It also silently removes real assets — a `.well-known` directory, a `.htaccess`-style file, a generated `_next`-style output directory — and does so without any diagnostic, because the pattern still matched plenty of other files. The cure is the `all:` prefix on the pattern, which turns the exclusion off for that subtree. It is worth applying deliberately rather than reflexively: `all:` on a directory that also contains version-control metadata will embed that too. ## Cause three: the file was not there when it was built If a pattern matches nothing at all, the build fails outright — that is the friendly case. The unfriendly one is a pattern that matches *some* files: an asset that exists only on a developer machine because it is untracked or ignored, or one that a generation step produces and the pipeline runs in the wrong order, leaves a build that succeeds and a binary that is short a file. The check is to build from a clean checkout and see whether the walk output changes. ## Cause four: the directive is not a directive A directive with a space after the slashes, or separated from its variable by a non-comment line, is an ordinary comment. The variable then holds an empty filesystem and nothing complains. This produces a *completely* empty walk rather than a partial one, which is a useful signal: partial means pattern semantics, empty means the directive itself. ## What to change afterwards Most teams end up with two habits. First, prefer naming directories over globbing files, so adding an asset does not require touching the directive. Second, add the assertion test — walking the embedded filesystem in a unit test and checking the entries you cannot ship without. Both convert a silent runtime gap into a build-time or pipeline failure, which is the whole point of embedding in the first place.

  • Which entries does embedding a directory skip by default, and how do you keep them?
    Every entry whose name begins with `.` or `_`, at any depth of the subtree, is excluded. Writing the pattern with the `all:` prefix — `//go:embed all:web` — disables that exclusion for that pattern. Apply it deliberately, since it will also pull in version-control metadata or tool caches sitting inside the same tree.
  • Why does //go:embed web/*.js miss web/vendor/lib.js?
    Patterns follow `path.Match`, where `*` matches non-separator characters only, so it never crosses a `/`. There is no recursive wildcard. Naming the directory — `//go:embed web` — embeds the subtree recursively instead, which is also why directory patterns are the more maintainable default: adding a file needs no directive change.
  • The build works on a laptop and fails in the pipeline with no matching files found. What is going on?
    The pattern is resolved against the package directory on the machine doing the build. A file that is untracked or ignored exists locally and not in a clean checkout, and a generated asset exists only if the generation step ran first. Fix the ordering, or commit the asset, so the same inputs are present in both places.
  • The walk prints nothing at all rather than a partial tree. What does that suggest?
    Partial results point at pattern semantics; a completely empty filesystem points at the declaration. Check that the directive has no space after the slashes, that only blank lines and line comments separate it from the variable, and that the variable is at package scope. A mis-typed directive is an ordinary comment and produces no error.

saying these in an interview costs you the question

  • Assumes a star pattern recurses into subdirectories
  • Thinks a redeploy can change what a binary embedded
  • Blames the runtime for a build-time selection
  • Lists the source tree instead of the embedded filesystem
  • Believes dot-prefixed files are embedded like any other