skip to content

In a cloud-drive service, why does each synced device ask for changes since its saved cursor instead of re-listing every file?

level: juniorimportance: must knowfreq 70%

answer

  1. cost proportional to changes, not files
  2. append-only log on the server
  3. one bookmark per device
  4. position assigned at commit time
  5. save the bookmark after applying

basics

~20 s

A server-side change journal records every change in order with a monotonic position. Each device stores the last position it applied, so one small request returns exactly what it missed instead of scanning and diffing the whole file tree.

solid answer

~50 s

The server appends every create, edit, move and delete in the account to a **change journal**, and each entry gets a monotonically increasing position. A device keeps an opaque `cursor` (the last position it applied) and calls something like `list_changes(cursor)`, which returns the entries after it plus a new cursor. The cost is proportional to what changed, not to how many files exist: a phone holding 200,000 files that missed three edits downloads three entries. Re-listing costs a full scan and a client-side diff on every sync, and a diff of two listings cannot easily tell a move from a delete plus a create, or a server-side delete from a local file not yet uploaded. The device saves the new cursor only after applying the batch, so a crash replays entries, which is why applying an entry must be idempotent.

code

json · 8 lines
json
{
  "entries": [
    {"position": 48214, "fileId": "f_91", "path": "/notes/plan.txt", "op": "edit", "rev": 7, "hash": "sha256:9c1e..."},
    {"position": 48215, "fileId": "f_42", "path": "/archive/q2.pdf", "op": "move", "from": "/q2.pdf", "rev": 3}
  ],
  "cursor": "opaque-token-for-48215",
  "hasMore": false
}

go deeper

for a junior

Remember the two pieces: a server-side log of changes in commit order, and a per-device bookmark into it. Be ready to say why the cost follows changes, not file count.

for a middle

Explain the loop precisely: fetch after cursor, apply, then persist the cursor, and why that order turns a crash into an idempotent replay rather than a gap.

for a senior

Show you have seen timestamp-based sync miss late-committing writes, and discuss retention, pagination for long-offline devices, and why the cursor is opaque.

for a principal

Frame the journal as the contract between server and every client version: its entry schema, retention window and cursor format are long-lived decisions that shape reset cost and future sharding.

## The problem a sync protocol solves A **cloud-drive service** keeps one account's files consistent across several devices: a laptop, a phone, perhaps a tablet. Each device holds a local copy of the tree and needs to learn, as cheaply as possible, what changed on the server since it last looked. The naive approach is to download the full listing of every file and compare it to the local tree. That works for a hundred files and collapses for a few hundred thousand: every sync costs a full scan on the server, a large download, and a diff on a phone with a limited battery and a metered connection. ## The change journal The standard answer is a **change journal** (also called a change log or event log) kept on the server per account, or per shared namespace. Every committed mutation appends one entry: - **what** changed: a stable file ID and its path; - **how** it changed: created, content edited, moved, deleted; - the **new revision** and a content hash, so the device can fetch the right bytes; - a **position** that increases monotonically within the journal. The journal holds metadata about changes, not file contents. The device downloads content separately, only for entries it needs. ## The per-device cursor A **cursor** is the device's bookmark into the journal: the position of the last entry it has applied. The sync loop is short: 1. The device calls `list_changes(cursor)`. 2. The server returns the entries after that position (paginated if many) and a new cursor. 3. The device applies each entry to its local tree. 4. Only after applying the batch does it persist the new cursor. Because each device keeps its **own** cursor, a laptop that synced a minute ago and a phone that was off for a week read different slices of the same journal. Neither needs to know about the other. Step 4 matters. If the device saved the cursor first and then crashed, it would skip the entries it never applied. Saving it last means a crash causes a **replay** instead, so applying an entry must be **idempotent**: applying edit r7 twice leaves the file at r7. Servers usually make the cursor **opaque**, an encoded token rather than a bare integer, so they can later change what it contains (a shard identifier, several positions) without changing clients. ## Why not re-list, and why not timestamps | Approach | Cost per sync | Correctness problems | |---|---|---| | Re-list whole tree and diff | Proportional to total files | Moves look like delete plus create; a file missing on the server is ambiguous | | Ask for files modified since time T | Proportional to changes, if indexed | Clock skew, writes that commit late with an earlier timestamp, ties within the same instant | | Journal plus cursor | Proportional to changes | Needs retention policy and idempotent apply | Timestamps look like a cheap cursor but are unsafe. Server clocks drift between machines, and a write that started before T can commit after the device queried, carrying a timestamp earlier than T, so the device never sees it. A journal position is assigned at commit time in commit order, so nothing can appear behind a cursor after the device has read past it. A diff of listings also loses **intent**. If `report.txt` disappears from one folder and appears in another with the same content, was it moved, or deleted and re-created? The journal records a move explicitly, so the device can move its local file instead of deleting and re-downloading it. ## Operating the journal - **Retention**: journals cannot grow forever. The server compacts old entries, and a device returning with a cursor older than the retained window must do a full resync. - **Pagination**: a device that was offline for a month may have thousands of entries to read; the server returns them in pages, each with its own continuation cursor. - **Ordering**: within one journal, positions are strictly ordered, so the device applies entries in the order they committed. - **Cheap no-op**: when nothing changed, the call returns an empty page and the same cursor, which is what makes frequent syncing affordable. ## Summary The journal turns sync from a scan into a read of only the deltas. The per-device cursor lets every device progress independently, and persisting it after applying the batch turns crashes into harmless replays rather than silent gaps.

  • Why do servers make the cursor an opaque token rather than a plain number?
    An opaque token lets the server change what the cursor contains without touching clients. It may later encode a shard, a journal generation, or one position per shared folder. It also stops clients from inventing cursors, such as jumping ahead to skip entries, and lets the server detect cursors it no longer recognises and answer with a clear reset signal instead of wrong results.
  • Does the change journal carry the file bytes?
    No. An entry carries metadata: file ID, path, operation, new revision and a content hash. The device decides which entries need content and fetches those bytes through a separate download path. Keeping bytes out keeps the journal small, fast to page through, and cheap to retain for a long window.
  • Why must applying a journal entry be idempotent?
    The device saves its cursor only after applying a batch, so a crash midway makes it re-read entries it may already have applied. If applying edit r7 twice left the same result as applying it once, the replay is harmless. Entries that carry the target revision, rather than a relative change, make this easy to guarantee.

saying these in an interview costs you the question

  • Just re-list the whole folder tree every few minutes and diff it
  • Use the device's local clock time as the sync cursor
  • Save the new cursor before applying the batch to save a step
  • A move can always be inferred reliably from two directory listings
  • The change journal should store the full contents of every file