How does a custom Airbyte source emit cursor state so the next sync resumes correctly?

## What state is in the protocol A `read` may receive `--state`, and it emits `STATE` messages as it goes. The platform stores the most recent STATE message a sync produced and passes it back on the next run. Modern connectors emit per-stream state: a state message carrying a `stream_descriptor` naming the stream and a `stream_state` blob whose contents are entirely the connector's business — `{"updated_at": "2024-03-01T10:00:00Z"}`, an id high-water mark, whatever the source lets you resume from. The framework never interprets it. That freedom is why state bugs are the most common class of custom-connector defect: nothing validates that your blob means what you think. ## Declaring the cursor in Python A stream declares `cursor_field`, the record field the state tracks. Older connectors implemented `get_updated_state(current_stream_state, latest_record)`, returning the merged state after each record. Newer ones implement `IncrementalMixin`, exposing `state` as a property with a getter and setter: the setter receives the previous run's blob at start-up, and the getter is read by the framework whenever it wants to checkpoint. Either way the arithmetic is the same — take the maximum of the stored cursor and the value on records you have already emitted. ## The ordering rule that actually matters A STATE message is a promise about the records that precede it in the output stream. The platform will, on the next run, start from that state and never re-read anything the state claims is done. So the only safe sequence is: emit records, then emit the state covering them. Two ways teams break it: **Advancing state before yielding.** If you compute the maximum cursor over a page and update state, then yield the page's records, a crash between the two loses that page permanently — the next sync starts after records that were never emitted. **Advancing state on unordered data.** If the API returns records in arbitrary order and you take the running maximum, a checkpoint mid-page can record a cursor higher than records still to come. Resume from it and those laggards are skipped. The fix is either to request a sorted read from the source (order by the cursor field ascending) or to only advance state at a boundary you can prove is complete — the end of a time slice, for instance. ## Checkpoint frequency If state is emitted only when a stream finishes, a sync that fails at hour five re-does five hours of work and re-hammers the API's rate limit. `state_checkpoint_interval` tells the framework to emit STATE every N records, giving you bounded rework. The trade is that more frequent checkpoints mean more state writes and, with an append destination, a longer tail of partially-loaded data on failure. Reasonable connectors checkpoint on the order of thousands of records, or once per completed slice. Slicing interacts here. `stream_slices` (or a declarative `step` on the cursor) splits a large read into windows — a month at a time on a two-year backfill. Each slice is a natural, provably-complete checkpoint boundary, which is why slicing plus per-slice state is the pattern that makes long backfills survivable. ## The declarative equivalent A manifest stream gets a `DatetimeBasedCursor`: `cursor_field` names the record field, `datetime_format` says how to parse it, `start_datetime` gives the floor for a first run, `step` sets the window size, `cursor_granularity` sets the smallest unit the source distinguishes, and `lookback_window` deliberately re-reads a margin behind the stored cursor. The cursor's value is injected into requests through start-time and end-time request options, so each slice becomes a filtered API call. The framework handles checkpointing per slice. ## Inclusive bounds, duplicates and the primary key Most sources filter with `>=`, and a cursor with second granularity cannot distinguish rows written in the same second. Both mean the boundary record set is re-read on the next run. That is the right default — losing rows is worse than repeating them — but it means an incremental append destination accumulates duplicates unless the stream declares a primary key and the user picks a deduplicating sync mode. A connector that reports no primary key on a stream it expects to be deduplicated has shipped a bug. `lookback_window` is the deliberate version of the same idea, for sources where rows can be written with a timestamp slightly in the past — you accept extra re-reads to stop losing late arrivals. ## Symptoms to recognise - **Every sync is a full sync.** State was never emitted (no checkpoint interval and the stream never completes), or the setter ignores the incoming state, or the cursor field is absent from records so the maximum stays null. - **Rows go missing after a failure.** State advanced ahead of emitted records, or advanced on unordered data. - **Duplicates every run at the boundary.** Expected with inclusive filters; the fix is a primary key and dedupe, not narrowing the filter. - **State goes backwards or resets.** A schema or cursor-field change altered the blob's meaning; treat cursor changes as a reset, not a silent upgrade.