What are Data Contracts in Confluent Schema Registry, and how do CSFLE and migration/data rules fit in?

## From schema to Data Contract Classically a registry entry was just a schema (Avro/Protobuf/JSON). A **Data Contract** extends that into a richer governance artifact attached to the subject's schema, comprising: - **Tags** — labels on fields/records (e.g. `PII`, `SENSITIVE`), usable by lineage/catalog and by rules. - **Metadata** — arbitrary key/value properties (ownership, sensitivity, etc.), with optional sensitivity marking. - **Rule sets** — ordered rules executed during serialization/deserialization. ## Rule categories 1. **Data quality / domain rules** — validate field values, often expressed as **CEL (Common Expression Language)** boolean expressions (e.g. `size(message.name) > 0`). On violation the rule can fail the message or route it (e.g. to a DLQ) via the rule's `onFailure` action. 2. **Transformation rules** — modify field values in flight (e.g. CEL_FIELD transforms). 3. **Migration rules** — `UPGRADE` / `DOWNGRADE` transformations that convert payloads between **compatibility groups**. They let you make an otherwise **breaking** change (a new major version) while older and newer clients keep interoperating: a consumer reading an old-version record can apply the UPGRADE rule to read it as the new shape, and vice versa. This is the mechanism for incompatible evolution without a synchronized cutover. 4. **Encryption rules → CSFLE**. ## CSFLE — Client-Side Field Level Encryption **CSFLE** encrypts specific **tagged fields** on the **client** (producer) before serialization output leaves the process, and decrypts on the consumer — so the **broker never sees plaintext** for those fields. Mechanics: - Tag the sensitive fields (e.g. tag `PII`). - Attach an **encryption rule** that targets fields with that tag and references a **KEK (key encryption key)** in a **KMS** — AWS KMS, GCP KMS, Azure Key Vault, HashiCorp Vault, or a local key for testing. - The serializer generates/uses a **DEK (data encryption key)**, encrypts the tagged fields (envelope encryption), and only authorized consumers (with KMS access) can decrypt. - Because it's **field-level**, non-sensitive fields stay queryable/cleartext while PII is protected end-to-end, satisfying regulations even from the broker operator. ## How rules execute Rules live in the contract's rule set and are invoked by the **serializer/deserializer** (the Schema Registry serdes run the rule engine). Producers run write-path rules (validation, encryption, transforms); consumers run read-path rules (decryption, migration upgrades, validation). ## Edge cases & design considerations - **Key management**: losing the KMS key means losing access to encrypted historical data; rotation and access policy are core design concerns. - **Migration rules** require defining **compatibility groups** (via a metadata property the compatibility level keys on) so the registry knows which versions a rule bridges. - **Performance**: per-message client-side crypto and rule evaluation add CPU/latency; DEK caching mitigates KMS round-trips. - **Searchability**: encrypted fields are opaque to brokers/ksqlDB — you can't filter/join on them server-side. - Data Contracts/CSFLE are a **client + registry** feature; the broker is unaware, which is why it works with vanilla Kafka brokers. ## Why it matters Data Contracts move governance (quality, evolution, privacy) into the schema artifact itself and enforce it at the edges via serdes — giving end-to-end PII protection (CSFLE), enforceable quality, and a path to breaking changes (migration rules) without coordinated downtime.