What do Airbyte's connector acceptance tests check before a connector ships?

## What they are for A connector's unit tests check your parsing code. The acceptance tests check that the artefact the platform will actually run behaves like a connector: the same suite runs against every connector in the ecosystem, driving the built Docker image through the four protocol commands with real credentials against a real (usually sandbox) account. They are configured per connector in `acceptance-test-config.yml`, which points at the image, the secret config, and the fixture files each suite needs, and they are run through the repository's connector tooling rather than invoked directly. ## The suites, mapped to the protocol **Spec.** The connector's specification must be valid JSON Schema, must not use constructs the platform cannot render, and must mark every credential as a secret so it is masked in the UI and redacted from logs. The spec is also checked for backward compatibility against the last published version, because it is a user-facing form: silently renaming a required field breaks every existing configured source. **Connection.** `check` must return success with the real config and failure with a deliberately broken one — typically an `invalid_config.json` fixture with a bad credential. This catches the very common bug of a `check` that always returns success because it calls an unauthenticated endpoint or swallows exceptions. **Discovery.** The catalog must be structurally valid: every stream has a JSON schema, declared sync modes are coherent, and incremental streams declare a cursor. Discovery is also subject to a backward-compatibility check against the previously released catalog, so a change that alters a field's type is surfaced as a breaking change rather than discovered downstream when a destination refuses the data. **Basic read.** The connector reads the configured streams and every record is validated against the schema its stream declared. This is the test that catches the classic mismatch where the schema says a field is an integer and the API returns a string, or where a documented field never actually appears. Optionally the suite compares output against a pinned `expected_records.jsonl` fixture, which asserts exact content rather than mere schema conformance. **Full refresh.** Two sequential full reads must produce the same records. A failure here means the read is nondeterministic — unstable ordering combined with a strict comparison, unfiltered volatile data, or paging that skips rows. **Incremental.** Two checks matter. Given an `abnormal_state.json` holding a cursor value far in the future, the connector should return no records — proving it actually applies the state to the source query rather than ignoring it. And a read starting from the state emitted by a previous read should return strictly less data than a read from scratch, proving state genuinely narrows the extraction. ## Where they get flaky, and how to keep them honest The suite runs against a live account, so anything that changes on the vendor's side moves the result. Pinned expected records rot as sandbox data drifts; timestamps, computed fields and vendor-assigned ids differ every run; ordering is rarely stable across pages. The configuration lets you relax the comparison — not requiring exact ordering, and excluding volatile fields from it — and the discipline is to pin a stable, dedicated test account whose data nobody edits. The alternative failure mode is worse: teams that fight flakiness by disabling suites end up shipping connectors whose incremental behaviour is untested, which is exactly the behaviour most likely to be wrong. Credentials are the other operational cost. The suite needs real secrets, so the connector's test config lives in a secret store rather than the repository, and a connector whose vendor has no free sandbox is hard to test honestly. ## What they do not catch Acceptance tests prove conformance, not correctness of meaning. They will not tell you that your cursor advances past unordered records, that you checkpoint before emitting, that your pagination skips rows under concurrent writes, or that a field you mapped means something different from what the analyst assumes. Those need targeted unit tests and, for the state bugs specifically, a deliberate mid-sync failure test. Treat the acceptance suite as the floor every connector must clear, and your own tests as the part that covers the logic you actually wrote.