A nightly rsync job over SSH copies the entire dataset every run, although almost nothing changes on the source. What rule does rsync use to decide a file needs transferring, and what are the usual reasons it keeps deciding yes?

## The rule rsync's default selection test is deliberately cheap and is called the *quick check*: a file on the destination is considered up to date if its **size** and its **modification time** both match the source. If either differs, the file is queued for transfer. Nothing is read, no checksum is computed — this is a metadata comparison, which is why rsync can walk millions of files quickly. Everything else follows from that one sentence. A job that re-transfers unchanged data is not a bug in the algorithm; it is a report that mtimes are not matching. ## Diagnose before you theorise Run the job with `-n -i` (`--dry-run --itemize-changes`) and read the change strings. The format is a per-attribute string such as `>f.st......`: - position 1: the action (`>` received, `<` sent, `c` created, `.` no change) - position 2: the file type (`f` file, `d` directory, `L` symlink) - then one letter per attribute that differs: `c` checksum/content, `s` size, `t` time, `p` permissions, `o` owner, `g` group `>f..t......` means the content is fine and only the timestamp differs — that is the fingerprint of this whole failure mode. `>f+++++++++` means the destination has no such file at all, which points somewhere else entirely (wrong path, wrong slash, destination being wiped between runs). Reading this string first saves an hour of guessing. ## Cause 1: the flags do not preserve times `-r` alone gives you recursion and nothing else. Times are preserved by `-t`, which is bundled into `-a`. Without it every file lands with the time it was written at the destination, which is by definition *now*, so on the next run all mtimes differ and the whole tree transfers again. The same happens if someone "tidied" a command into `-rlpgoD` (archive minus `-t`) or added `--no-t` to `-a`. This is the single most common answer, and it is worth stating the causality out loud: `-a` is not just about fidelity, it is what makes repeat runs cheap. ## Cause 2: the destination cannot hold the timestamp Some filesystems store coarser times than Linux does. FAT and exFAT round to two seconds; some SMB/CIFS mounts and some object-store gateways behave similarly, and a few network filesystems apply their own clock. rsync then writes the right value and reads back something slightly different, so the comparison fails forever. The fix is `--modify-window=NUM`, which tells rsync to treat times within NUM seconds of each other as equal — `--modify-window=1` is enough for FAT's two-second granularity, and `--modify-window=2` is a common belt-and-braces setting. A related variant: a destination mounted with an option that ignores utimes, or a container/NAS layer that stamps its own times. Same symptom, same fix, but confirm with `stat` on both sides before reaching for the flag — you want to see the actual pair of timestamps, not assume. ## Cause 3: the source really is changing A build that regenerates identical bytes, an export that rewrites a whole directory nightly, or a `cp` step upstream that resets times all produce files whose content is unchanged but whose mtime is new. rsync is behaving correctly; the producer is lying about change. Two options: - **`-c` (`--checksum`)** replaces the quick check with a full-file checksum comparison on both sides. It is correct and it is expensive: both hosts read every byte of every file before anything is transferred. On a large dataset over a nightly window this can cost far more than the redundant transfer it avoids. Use it deliberately. - **Fix the producer** so unchanged outputs keep their timestamps, or sync from a source of truth that does not rewrite. This is usually the better engineering answer. ## The inverse trap The mirror image of this failure is a job that transfers *too little*. `--size-only` compares size alone and ignores mtime entirely; it is sometimes reached for as a "fix" for re-transfer problems, and it silently misses every same-size edit — a flipped byte in a fixed-length record, an image edited in place, a config value swapped for another of equal length. If you find `--size-only` in an existing job, treat it as a defect to justify, not a setting to keep. `-I`/`--ignore-times` is the opposite extreme: it disables the quick check and transfers everything, which is occasionally useful for a one-off forced refresh and disastrous as a permanent setting. ## Checklist for the incident 1. `rsync -ni ...` and read the itemise string — is it `t`, `s`, `c`, or `+++++++++`? 2. If `t`: is `-t`/`-a` present? `stat` the same file on both sides and compare timestamps. 3. If the timestamps differ by a sub-second or two-second amount: `--modify-window`. 4. If content genuinely changed: look upstream at what rewrites the files. 5. Only then consider `-c`, and measure what the full read costs on both hosts.