skip to content

Why do archives whose entry names came from filepath.Join on Windows extract as flat files on Linux?

level: seniorimportance: should knowfreq 30%

answer

  1. the separator became part of the data
  2. one platform's choice leaked outward
  3. a backslash is a legal Linux filename byte
  4. convert at the boundary with ToSlash

basics

~20 s

Because the Windows build joined entry names with backslashes, and a backslash is an ordinary character in an archive name, so an extractor elsewhere creates one file literally called docs\index.html. Convert operating-system paths with filepath.ToSlash before storing them.

solid answer

~50 s

Archive entry names are slash-separated by definition, on every platform. A packaging tool that walks the source tree gets operating-system paths, and on a Windows build `filepath.Rel` and `filepath.Join` produce `docs\index.html`; writing that straight into the header stores a name whose only element is the literal string `docs\index.html`. A Windows extractor may be forgiving, but a Linux one creates a single oddly named file in the destination directory instead of a `docs` directory. The fix is one conversion at the boundary: compute the name relative to the root with `filepath.Rel`, then `filepath.ToSlash` it before it becomes an entry name, and `filepath.FromSlash` on the way back into operating-system space. An absolute Windows source path also carries a volume — `filepath.VolumeName` reports `C:` for a path rooted there and an empty string on Unix — and a volume must never reach a stored name.

code

go · 6 lines
go
rel, err := filepath.Rel(root, osPath)
if err != nil {
	return err
}
// rel is "docs\index.html" on Windows, "docs/index.html" on Linux
hdr.Name = filepath.ToSlash(rel)

go deeper

for a junior

Know that a name on disk uses the platform's separator while an archive or wire name always uses a slash, and that filepath.ToSlash is the conversion between the two.

for a middle

Explain why the two builds diverge: filepath.Rel and filepath.Join follow the compiled-for platform, so the backslash becomes part of the stored data unless something converts it at the boundary.

for a senior

Walk the diagnosis end to end — reproduce on a Windows runner, or write the test that asserts produced entry names contain no backslash — and then place a single conversion point instead of patching each writer.

for a principal

Decide what the tool actually guarantees to Windows users and what that guarantee costs: a build matrix with a real Windows runner and someone to keep it green, or a narrower support claim you can honour.

## The bug in one sentence A separator is a **serialisation choice**, and this tool let the platform it happened to be compiled for decide that choice for data that other platforms read. ## How it happens A cross-platform packaging tool ships as one binary for Linux, macOS and Windows. It walks a source tree, and every name it gets back is an operating-system path: `docs/index.html` on Linux, `docs\index.html` on Windows. It computes each entry's name relative to the root and writes it into the archive header. On Linux everything is right, because the operating-system separator and the archive's separator are the same character. On Windows the header now contains `docs\index.html`. The archive format does not carry a "which platform wrote this" flag that a reader could compensate with; the name is simply a string, and the reader splits it on slashes. There are none. So the reader sees a single element whose name happens to contain a backslash — a perfectly legal character in a Linux file name — and creates one flat file. Every directory in the tree is gone, and if two entries differ only inside their (now flattened) directory part, they collide. ## The fix: convert once, at the boundary ```go rel, err := filepath.Rel(root, osPath) if err != nil { return err } hdr.Name = filepath.ToSlash(rel) ``` `filepath.ToSlash` replaces the platform separator with a slash. On Linux and macOS it returns the string unchanged, because the separator is already a slash — which means you can and should call it unconditionally: it costs almost nothing, and it documents that the value is leaving operating-system space. `filepath.FromSlash` is the mirror image, converting a slash-separated name into an operating-system one on the way back in. What you must **not** do is a manual string replacement of backslashes with slashes. On Unix a backslash is a legal character inside a file name, so a hand-rolled replacement silently corrupts real names, while `ToSlash` only ever rewrites the separator of the platform it was compiled for. ## The volume, the other Windows-only surprise An absolute Windows path begins with a volume: a drive letter such as `C:` or a UNC prefix. `filepath.VolumeName` reports that prefix, and returns the empty string on a Unix build. Any code that produces a stored name by trimming a prefix by hand — slicing off `len(root)` bytes, say — can leave a volume, a leading separator, or a half-eaten element in the result. Computing the name with `filepath.Rel` against the root avoids all of that, and asserting in a test that `filepath.VolumeName` of every produced name is empty is a cheap guard. ## Why the tests did not catch it Because on Linux the missing conversion is invisible: `filepath.Rel`, `filepath.Join` and `filepath.ToSlash` all agree there. Two things change that. **A build matrix with a real Windows runner.** The tool ships to Windows users, so the tests have to run where those users are; cross-compiling with `GOOS=windows` proves the code builds and lets `go vet` run, but a binary built for another platform cannot be executed on the build machine, so it proves nothing about behaviour. **A platform-independent unit test on the names themselves.** Build the entry list from a fixture tree and assert two properties: no produced name contains a backslash, and the sorted list matches a golden list of slash-separated names. That test fails on Windows and passes on Linux — which is the point, since it is the Windows job in the matrix that has to go red. A companion test can construct names from synthetic Windows-shaped input and assert the conversion function's behaviour directly, which fails everywhere and gives faster feedback. ## The general rule the incident should leave behind Draw a line through the program: - **inside** the line, operating-system paths, built with `filepath` and never stored; - **outside** the line — archive entries, manifests, databases, anything on the wire — slash-separated names, built with the `path` package and never handed to `os.Open` directly; - **at** the line, exactly two functions, `filepath.ToSlash` outbound and `filepath.FromSlash` inbound. The maintainer's version of this is blunter: a name that leaves the process is data, and data does not get to depend on which machine produced it.

  • Why does this bug never appear in a Linux-only CI pipeline?
    On Linux the platform separator is already a slash, so `filepath.Rel`, `filepath.Join` and `filepath.ToSlash` all produce the same string and the missing conversion is invisible. You need a Windows runner in the matrix, or a unit test that feeds Windows-shaped names through the conversion and asserts the result contains no backslash — the second fails on every platform, so it gives faster feedback.
  • What does filepath.ToSlash do on Linux, and is calling it there wasteful?
    It returns the path unchanged, because the separator it would replace is already a slash. Calling it unconditionally is the right habit: the cost is negligible and it documents at the call site that the value is leaving operating-system space. `filepath.FromSlash` is the mirror for names arriving from an archive, a manifest or the network.
  • Why compute the entry name with filepath.Rel rather than trimming the root prefix by hand?
    Because hand-trimming leaves platform debris: a leading separator, a half-eaten element when the root has no trailing separator, or a Windows volume such as the `C:` that `filepath.VolumeName` would report. `filepath.Rel` produces a relative name lexically or returns an error — for example when the two names sit on different volumes — which is a failure you want reported rather than silently packaged.

saying these in an interview costs you the question

  • Says Windows accepts slashes, so no conversion is needed
  • Replaces backslashes with a string replacement by hand
  • Thinks the archive format records the writing platform
  • Tests only on Linux and calls the tool cross-platform
  • Uses filepath.Join to build the stored entry name