skip to content

What does each Airbyte sync mode do to the destination table?

level: juniorimportance: must knowfreq 76%

answer

  1. two halves: how to read, how to write
  2. replace, pile up, add new, collapse
  3. a cursor field drives the incremental read
  4. the deduped variant also needs a primary key
  5. set per stream, not per connection

basics

~20 s

Airbyte pairs a read mode with a write mode. Full Refresh Overwrite replaces the table each sync; Full Refresh Append re-appends everything; Incremental Append adds only records past the cursor; Incremental Append + Deduped keeps one current row per primary key.

solid answer

~50 s

An Airbyte sync mode is written `read | write`, and it is chosen **per stream**, not once per connection. - **Full Refresh | Overwrite** — the source re-reads the whole stream and the destination table ends up holding exactly that snapshot; the previous contents are replaced. - **Full Refresh | Append** — the whole stream is read again and added on top, so the table grows by a full copy every run. - **Incremental | Append** — the source reads only records whose *cursor field* (something like `updated_at` or an autoincrementing id) is at or past the value saved from the previous sync, and appends them. Updated rows appear as extra versions, and boundary records can repeat. - **Incremental | Append + Deduped** — the same incremental read, but the destination then rebuilds a final table holding one row per primary key, keeping the newest by cursor. The UI only offers combinations that both the source and the destination declare support for.

code

text · 7 lines
text
source now: (1, alice, updated 10:00), (2, bob, updated 12:00)
previous sync saw: (1, alice-old, updated 09:00)

Full Refresh | Overwrite   -> 1 alice(10:00), 2 bob(12:00)
Full Refresh | Append      -> 1 alice-old(09:00), 1 alice(10:00), 2 bob(12:00)
Incremental | Append       -> 1 alice-old(09:00), 1 alice(10:00), 2 bob(12:00)
Incremental | Append+Dedup -> 1 alice(10:00), 2 bob(12:00)

go deeper

for a junior

Be ready to name the four mode pairings and say in one sentence what each leaves in the destination table. Knowing that a cursor field drives the incremental read is expected.

for a middle

Explain why the pairs exist: read mode is the source's job, write mode the destination's, and only combinations both sides support are offered. Be precise that dedup happens in the destination, after the append.

for a senior

Show you pick modes per stream from source characteristics — mutability, volume, whether a trustworthy cursor exists — and that you know Incremental | Append shifts the latest-row problem downstream instead of solving it.

for a principal

Own the standard: which stream shapes get which mode across dozens of connections, what full-refresh reloads cost the source system and the warehouse, and how to keep those defaults enforceable rather than per-engineer taste.

## A sync mode is two decisions Every stream inside an Airbyte connection carries its own sync mode, shown as `Read mode | Write mode`. The left half decides what the **source** hands over on each run; the right half decides what the **destination** does with what arrived. They are independent decisions, which is why the modes read as pairs, and why one connection can sync a small lookup table one way and a large event table another. The read side has two values. **Full Refresh** re-reads the entire stream every time — every row of the table, every page of the API. **Incremental** re-reads only the records whose *cursor field* is at or past the cursor value Airbyte saved at the end of the previous sync. The cursor field is a column the connector declares (a *source-defined cursor*) or that you pick in the stream settings; it must be something that moves forward as rows change, typically `updated_at` or a monotonic id. The write side has three values: **Overwrite**, **Append**, and **Append + Deduped**. In the Airbyte protocol these are the destination sync modes `overwrite`, `append` and `append_dedup`. ## Full Refresh | Overwrite The destination ends the sync holding exactly what the source produced this run and nothing else. Destinations implement it as a load into a temporary location followed by a swap, so the final table is not left half-empty mid-sync. This is the self-healing mode: whatever drifted, was deleted, or was mangled at the source is corrected on the next run, because the previous state is discarded. Its cost is that you pay the full read every time and you keep no history. ## Full Refresh | Append The full read is added on top of what is already there. After ten syncs the table holds ten complete copies of the source, distinguishable only by the extraction timestamp Airbyte stamps on each row. This is useful when you deliberately want periodic snapshots — a slowly-changing reference table you want to audit over time — and it is a footgun when someone selects it expecting deduplication. Storage and query cost grow linearly with sync count. ## Incremental | Append Only records past the saved cursor are read, and they are appended. This is the cheapest mode for large append-mostly streams such as events or logs. Two properties surprise people. First, an updated row does not replace anything: you get a second row for the same business key, and downstream queries must pick the latest themselves. Second, the mode does not promise uniqueness — connectors generally re-read records *at* the last cursor value rather than strictly after it, so boundary records repeat, and a retried sync can re-append records the destination already accepted. ## Incremental | Append + Deduped The read is identical to Incremental | Append; the difference is the destination step. Records are appended to a raw table, then Airbyte builds a final table containing one row per **primary key**, keeping the version with the highest cursor value. This mode therefore needs both a cursor field and a primary key, and it is the one people mean when they say they want the warehouse table to mirror the source. Deletes are still invisible unless the source itself emits a deletion marker. ## Where the mode is set, and what constrains it Sync mode is a per-stream setting inside a connection. The source's catalog declares which read modes each stream supports — some API streams have no usable cursor at all and offer full refresh only — and the destination declares which write modes it supports. Airbyte offers the intersection, so 'incremental is greyed out' is usually a statement about the connector, not about your configuration. ## What the mode does not change Scheduling is separate: a connection runs manually, on a basic interval, or on a cron expression, and the sync mode says nothing about how often. The mode also does not change the fact that records land in Airbyte-managed raw storage first and are typed into the final table afterwards. Mode names have moved between versions — the deduped mode was previously presented as *Incremental | Deduped + History* and maintained an extra history table — so name the behaviour, not just the label, in an interview.

  • Is the sync mode a property of the connection or of the stream?
    Of the stream. A single connection can run one stream as Full Refresh | Overwrite and another as Incremental | Append + Deduped. The available choices come from the intersection of what the source declares per stream and what the destination supports, which is why incremental is sometimes unavailable for a particular stream.
  • Why can Incremental | Append leave duplicate rows even without any source updates?
    Connectors generally re-read records at the saved cursor value, not strictly greater than it, so records sharing the boundary timestamp are emitted again. A sync that fails after the destination committed some records and then retries can also re-append them. Append mode has no primary key to collapse either case.
  • When is Full Refresh | Append actually the right choice?
    When you want a dated series of complete snapshots — auditing how a small reference or configuration table looked over time, or reconstructing a history the source does not keep. You accept that the table grows by one full copy per sync and that every query must filter to a single extraction timestamp.

Overwrite is rephotocopying the whole document each morning; incremental append is adding only today's new pages to the stack; the deduped mode also throws away the superseded pages.

saying these in an interview costs you the question

  • Says incremental sync also removes rows deleted at the source
  • Claims Incremental | Append can never produce duplicates
  • Thinks Overwrite keeps prior syncs as history
  • Chooses the deduped mode without configuring a primary key
  • Assumes sync mode is set once per connection for all streams

context