Your custom Airbyte source keeps failing on HTTP 429 — how do you make it back off?

## 429 is normal traffic, not an exception Any connector that backfills a SaaS API will hit its rate limit; the limit exists precisely to stop what a backfill does. So rate limiting is part of a connector's steady-state behaviour, and "the sync fails on 429" means the connector was written as if the happy path were the only path. The design goal is to keep the sync alive at whatever throughput the vendor permits, and to fail loudly rather than quietly when it cannot. ## The Python CDK hooks A stream built on the CDK's HTTP base class decides retry behaviour in two places: a predicate that says whether a given response should be retried (the default already retries 429 and 5xx), and a method returning the number of seconds to wait before the next attempt, which receives the response so it can read headers. A maximum retry count bounds the loop. There is also a switch controlling whether a non-retryable error status raises or is passed through to your parsing code — useful for APIs that use 404 to mean "this sub-resource is empty", where raising would abort a stream that is actually fine. Newer CDK versions restructure this into an error-handler object with backoff strategies rather than method overrides, so check which shape your CDK version expects before writing overrides that are never called. ## The declarative equivalent A manifest requester takes an error handler. Its `response_filters` match on status codes, or on text in the body, and assign an action: retry, fail, ignore, or treat as rate limited. Its backoff strategies supply the delay: an exponential strategy with a factor, a constant strategy, or a strategy that reads a wait time out of a named response header — which is the one you want when the API sends `Retry-After` or a reset epoch. ## Respect the vendor's signal The difference between a connector that survives a backfill and one that does not is usually whether it reads the API's own headers. Many APIs return `Retry-After` in seconds, or a rate-limit reset expressed as an absolute timestamp, or remaining-quota headers you can use to slow down *before* being blocked. Using them beats guessing: exponential backoff either sleeps far longer than the actual window, turning a two-hour sync into an overnight one, or retries before the window resets and burns another attempt. Preemptive throttling — slowing when the remaining-quota header drops — is the more advanced version and turns a spiky connector into a well-behaved one. Also mind whether the limit is per second, per minute, per day or per credential. A daily quota cannot be waited out inside a sync; the correct response there is to fail cleanly after checkpointing and let the schedule pick it up, not to sleep for hours holding a worker. ## Failing well is part of the answer When the retry budget is exhausted, three things must be true. The failure must be surfaced as an error, not hidden — a connector that catches the 429 and returns the pages it managed is manufacturing silent data loss, and the destination will happily load an incomplete stream. The error should be classified so the platform and the user can tell a transient system problem from a configuration problem such as a plan-level quota that will never clear. And state must have been checkpointed as you went, so the next run resumes near where you stopped instead of replaying the whole backfill into the same wall. ## Reducing the pressure, not just absorbing it Retry logic is the last line. Ahead of it sit choices that determine how often you hit the limit at all: a larger page size means fewer requests for the same rows; narrower incremental windows mean you fetch less; avoiding one API call per parent record where a bulk endpoint exists can cut request counts by orders of magnitude; and concurrency, if the CDK version you use offers it, must be sized under the vendor's ceiling rather than at it. A connector that pages 25 rows at a time across a five-year backfill is rate-limited by its own design, and no backoff strategy repairs that. ## What interviewers listen for A weak answer is "add a sleep and retry". A strong one names the layer the retry belongs in, distinguishes reading the vendor's reset signal from blind exponential backoff, insists on failing rather than truncating, connects checkpoint frequency to the cost of a rate-limit failure, and mentions reducing request volume as the real fix for a chronically throttled backfill.