A colleague runs `sort data.txt > data.txt` in bash and data.txt comes back empty. Why does the file lose its contents, and what is the correct way to filter a file in place?

## The shell does the redirection, not the command The single most useful mental model here is that `>` is not an argument, an option, or something the program interprets. When bash runs a simple command it works through a fixed order: it splits the line into words, performs expansions, **removes the redirections from the word list and applies them**, and only then executes the program with the remaining words as its arguments. `sort` receives exactly one argument — `data.txt` — and inherits a descriptor 1 that is already attached to a freshly emptied `data.txt`. Because `>` opens the target with the `O_TRUNC` flag, the truncation happens at *setup* time. By the time `sort` calls `open("data.txt")` to read its input, the file it opens is the same, now zero-byte, file. There is no race, no timing subtlety and no version where this works: `cmd file > file` reliably destroys `file`. ```bash printf 'b\na\n' > data.txt sort data.txt > data.txt # data.txt is now empty ``` The same trap wears other costumes: `grep -v DEBUG app.log > app.log`, `tr -d '\r' file > file`, `jq '.' conf.json > conf.json`. Any filter whose input and output name the same path loses the input. ## Why the tool cannot save you Candidates sometimes propose "sort should detect this". It mostly cannot, because by the time it starts, the previous contents are already gone — the shell truncated the file in a process that has since exec'd into `sort`. A tool *can* detect that its input and output paths refer to the same file when it is asked to open both itself, which is precisely why GNU `sort` offers `-o`: ```bash sort -o data.txt data.txt # documented as safe: sort reads the input fully first ``` Here `sort` owns the output path, so it can read the input completely before opening the output for writing. That guarantee comes from `sort`, not from the shell. ## The correct patterns **1. Temp file plus rename.** The workhorse, and the only one that works with arbitrary tools: ```bash sort data.txt > data.tmp && mv data.tmp data.txt ``` The `&&` matters: if the filter fails, you keep the original instead of replacing it with a half-written file. A rename within the same filesystem is atomic, so any concurrent reader sees either the old file or the new one, never a truncated one. Create the temporary next to the target — not in `/tmp` — so the rename stays on the same filesystem; a cross-device `mv` degrades into a copy plus delete and loses atomicity. (Generating that temp name safely, and deleting it when the script dies, is its own discipline.) **2. A tool's own in-place mode.** `sed -i`, `perl -i`, `sort -o`, `sponge` from moreutils. Note that `sed -i` is not magic either: it writes a temporary file and renames it, which is why it changes the file's inode and can break hard links. And `sed -i` needs an argument on BSD/macOS (`sed -i '' …`) where GNU takes none. **3. Read it all first.** `data=$(cat data.txt)` then write — fine for small files, hopeless for large ones, and it mangles trailing newlines. ## noclobber as a guardrail Bash can refuse to overwrite an existing file: ```bash set -o noclobber # same as set -C date > data.txt # bash: data.txt: cannot overwrite existing file date >| data.txt # >| overrides noclobber for this one redirection set +o noclobber ``` Under `noclobber`, `sort data.txt > data.txt` fails loudly instead of silently emptying the file — a genuine save. Know its limits: it only affects `>` (not `>>`, and not tools that open files themselves), it is a per-shell option that scripts must set for themselves, and it is off by default in every shell you will meet. Treat it as a seatbelt for interactive sessions, not as a substitute for the temp-file-and-rename pattern in a script. ## What an interviewer is really testing This question separates people who have memorised operators from people who know *when* each stage of command execution happens. Say the order out loud — expand, redirect, execute — and the empty file explains itself. It also generalises: the same ordering is why a redirection error aborts the command before it runs at all, and why `> file` creates the file even when the command turns out not to exist.