Why is filepath.WalkDir cheaper than filepath.Walk over a large directory tree?

## The two signatures ```go func filepath.Walk(root string, fn filepath.WalkFunc) error type filepath.WalkFunc func(path string, info fs.FileInfo, err error) error func filepath.WalkDir(root string, fn fs.WalkDirFunc) error type fs.WalkDirFunc func(path string, d fs.DirEntry, err error) error ``` Everything else about them matches: both start at `root` and call the callback for `root` itself first, both descend in lexical order within each directory, both call the callback sequentially from one goroutine, and neither follows symbolic links — a symlink is reported as an entry and not descended into. The only difference is the second parameter, `fs.FileInfo` versus `fs.DirEntry`. ## Why that parameter is the whole story Reading a directory yields, per entry, a name and — on the platforms that support it — a type hint saying whether it is a directory, a regular file, a symlink and so on. That is exactly what an `fs.DirEntry` exposes: `Name()`, `IsDir()`, `Type()`. Nothing more had to be asked of the filesystem to answer them. An `fs.FileInfo` is a bigger promise: `Size()`, `Mode()` with permission bits, `ModTime()`, `Sys()`. None of that comes out of the directory read. To hand your callback one, `filepath.Walk` must issue a per-entry metadata call before it can call you — for every file and every directory in the tree, unconditionally. So on a tree with `D` directories and `N` entries: * `filepath.WalkDir` ≈ `D` directory reads (plus one stat of the root). * `filepath.Walk` ≈ `D` directory reads **plus N metadata calls**. For an asset indexer over a checked-out repository — a deep tree of very many small entries, most of which the indexer discards on the strength of the name or the extension alone — `N` dwarfs `D`, and almost every one of those metadata calls is thrown away. The standard library's own documentation for `Walk` says as much: it is less efficient than `WalkDir`, which avoids the per-entry call. ## What it looks like when you are wrong about it The symptom is not a crash. It is a walk whose wall-clock time scales with file count far worse than expected, and a CPU profile in which the hot path under your walk function is filesystem metadata work rather than your own filtering. This bites hardest on the engineer porting a tool from a language whose directory-walk API hands back full metadata for every entry as a matter of course: the natural translation reaches for `filepath.Walk`, because its callback signature is the familiar one, and the cost model quietly comes along. The same mistake survives a switch to `WalkDir` if the first line of the callback is `info, err := d.Info()`. That reintroduces exactly the per-entry call you just removed. The rule is to filter on `d.Name()`, `d.IsDir()` and `d.Type()` first, and reach for `d.Info()` only for the entries that survive the filter. ## When Walk is still fine If your callback genuinely needs `Size()` or `ModTime()` for **every** entry — a `du`-style disk-usage tool, say — then Walk's eager stat costs the same as WalkDir plus `d.Info()` everywhere, and there is no win. Even then, WalkDir is the better default: it keeps the choice in your hands, and it is the shape that generalises to `fs.WalkDir` over an abstract filesystem. ## Things that are not the difference * Neither function is concurrent. Both are a single sequential traversal, so your callback does not need to be safe for concurrent use, and neither will saturate a fast disk on its own. * Neither skips hidden files or any other class of entry for you; both report everything they find. * Neither follows symbolic links, so neither can be sent into a cycle by one. * `WalkDir` is not lazy in the sense of streaming results back — it is a push traversal in both cases, driven by the callback's return value. ## Practical shape ```go err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { if err != nil { return err } if d.IsDir() || filepath.Ext(d.Name()) != ".png" { return nil } info, err := d.Info() // paid only for the entries that matter if err != nil { return err } return record(path, info.Size()) }) ``` The error parameter is not decoration — handling it is a separate discipline, and returning it unconditionally stops the whole traversal.