Compare KMS-backed volume encryption versus client-side payload encryption for Kafka: which threats does each address, and when would you mandate both?

## Two layers, two threat models Kafka has **no native log-segment encryption**, so at-rest protection is added either **below** the broker (disk/volume) or **above** it (payload). They are not interchangeable — each defends a different attacker. ### KMS-backed volume / disk encryption (below Kafka) - **What:** the block device holding `log.dirs` is encrypted. Options: **AWS EBS encryption** with **KMS** keys, **GCP persistent disk with CMEK** (customer-managed encryption keys), **Azure disk encryption**, or self-managed **LUKS/dm-crypt**. A **KMS** (Key Management Service) holds the keys; the platform decrypts blocks on read. - **Transparent to Kafka:** no code changes, full functionality (compaction, filtering, schema validation all work because the broker sees plaintext after the volume decrypts). - **Threats addressed:** stolen or decommissioned **physical disks**, leaked **snapshots/backups**, and ticking the 'data encrypted at rest' compliance box. - **Threats NOT addressed:** a **compromised broker process**, a **malicious/over-privileged operator** with shell or filesystem access, or **memory scraping** — the data is plaintext in RAM and through the filesystem while the broker runs. Key access is also typically broad (the instance role can decrypt), so the blast radius of a host compromise includes the data. ### Client-side / application payload encryption (above Kafka) - **What:** producers encrypt values (often via **envelope encryption** with a KMS-issued data key + AES-GCM) before `send()`; consumers decrypt after `poll()`. The broker stores only **ciphertext**. - **Threats addressed:** the **broker and its operators**, anyone reading the **disk or memory** of the broker, and inter-region replication exposure. Enables **crypto-shredding** (destroy a subject's key to satisfy GDPR erasure when physical deletion is impractical). - **Costs / threats NOT addressed:** the broker can no longer **inspect content** — **value-based compaction**, **server-side filtering**, **ksqlDB/Streams value logic**, and **Schema Registry validation of ciphertext** all break (compaction by *key* still works since keys stay cleartext); **key management, rotation, and distribution** become first-class operational concerns; CPU and KMS round-trip overhead; and it does **not** protect the **endpoints** (a compromised producer/consumer still holds plaintext). ## Decision framework - **Low sensitivity, internal:** KMS volume encryption + TLS is usually enough and keeps full functionality. - **Regulated PII (PCI/HIPAA/GDPR):** mandate **both**. Volume encryption gives cheap, transparent defense-in-depth and satisfies at-rest mandates and physical-theft scenarios; client-side encryption removes trust in the broker operator and enables key-based erasure. Apply client-side encryption **selectively** (field-level on PII columns) to preserve as much broker functionality as possible. - **Multi-tenant / untrusted operations (e.g., managed cloud Kafka you don't control):** lean harder on client-side encryption because you can't trust the operator. ## Why both isn't redundant The layers cover **disjoint attackers**: physical/snapshot theft (volume) vs. live broker/operator compromise (payload). Removing either leaves a real gap. TLS adds the **in-transit** dimension, completing the in-transit / at-rest-physical / at-rest-logical triad. ## Pitfalls a principal should flag - Assuming KMS volume encryption protects against insiders (it doesn't). - Encrypting everything client-side and then discovering compaction/filtering/validation broke — scope encryption to sensitive fields. - Key-management gaps: rotation, caching, per-subject keys for crypto-shredding, and ensuring downstream stores (tiered storage, replicas, lakes) inherit the same protections.