How would you plan the initial CDC snapshot of a 4 TB production database that cannot absorb extra load?

## Frame the constraint before the plan Every cost in a large snapshot scales with elapsed time: log segments retained since the pinned position, superseded row versions the read view prevents reclaiming, read I/O stolen from the workload, and the exposure window in which a failure wastes everything done so far. So the first number to establish is **duration**, and the first invariant is that duration must sit well inside the source's log retention window with room for a stall. Get the number empirically. Scan one representative table with production-like pacing, measure rows per second and the effect on source latency, and extrapolate to 4 TB. Guessing here is what produces the hour-eleven failure. ## Decision one: does a source scan need to happen at all? The cheapest snapshot is one you do not run. - **Nobody needs history.** Cache invalidation, search indexing, audit streams and event-driven integrations often care only about the future. Capture schema only, start at the current log position, done. - **History already exists.** If a nightly full extract already lands in a warehouse, take the log position at or before that extract's cut, stream forward, and reconcile the extract by primary key. This is frequently the correct answer at 4 TB and is under-proposed in interviews. - **History is needed and lives only in the source.** Now you scan — but perhaps not all of it. Excluding blob columns, cold partitions and soft-deleted rows routinely removes most of the volume. ## Decision two: where the scan reads Reading from a **replica** removes the scan's I/O from the primary, which is often the whole point when the primary has no headroom. Two things must be checked: - **Log-position portability.** The position you pin must be interpretable against the stream you will consume. A global transaction identifier is a portable identity; raw file-and-offset coordinates are replica-local and mean something different on the primary. Getting this wrong anchors the handover to the wrong point and loses data silently. - **Engine support and replica health.** Some engines only allow logical decoding on a standby from recent versions, and a long scan puts the same version-retention pressure on the replica that it would have put on the primary — including the risk that the replica falls behind or has to cancel the query to keep applying changes. Reading from the **primary** with aggressive throttling is the simpler, less surprising option when the replica route is not clean. ## Decision three: how the scan is paced **Watermark-based incremental snapshot** is the strongest default at this size: chunked, resumable per chunk, no long read view, and — decisively — the log is consumed throughout, so nothing is retained and there is no catch-up cliff at the end. The prerequisite is a writable signalling table in the source, which is a political question as often as a technical one. If that is unavailable, use a chunked classic snapshot with checkpointing so a restart resumes at a chunk boundary, and parallelise across tables — or across key ranges of the largest table — up to the source's I/O tolerance. Parallelism converts directly into reduced duration, and duration is the thing every risk here is proportional to. Past a point it becomes the load you promised not to add, so pace it against a source-latency SLO and be willing to run over days. ## Decision four: what the sink must guarantee None of this is safe unless the sink applies changes **idempotently and in per-key order**. That single property makes seam overlap harmless, makes mid-backfill restarts free, and makes a targeted re-seed of one table a routine operation rather than an incident. If the sink cannot promise it, fix the sink before planning the snapshot — otherwise every strategy above degrades into a one-shot operation you must get right the first time. ## Sequencing and guardrails Order the work: prove the sink's idempotency, raise log retention and disk headroom, instrument retained log bytes, slot lag, oldest-transaction age and source latency, run a pilot on one large table, extrapolate, then start with an explicit abort criterion. Publish what consumers see during the backfill — a sink that is complete for some key ranges and empty for others is a correct intermediate state, but only if downstream knows. ## The tradeoff to own out loud You are trading **source impact against convergence time**, under a hard ceiling set by log retention. Going fast risks the OLTP workload; going slow risks the retention window and leaves consumers on partial data for longer. The lead's job is to name which of those the business can absorb, size the headroom explicitly, and choose the strategy that keeps a failure costing one chunk rather than one week.