Does Apache Kafka natively encrypt the data it writes to disk (log segments)? If not, how do teams achieve encryption at rest?
answer
- No native log-segment encryption
- LUKS / KMS-backed EBS = disk layer
- Client-side payload = above broker
- TLS = in transit only
- log.dirs files are plaintext
basics
~20 sNo. Open-source Kafka has no built-in encryption of log segment files on disk. Encryption at rest is provided externally: encrypt the broker's volume/disk (LUKS or a cloud KMS-backed encrypted EBS volume), or encrypt the message payload before producing it.
solid answer
~40 sApache Kafka does not encrypt its on-disk log segments natively; the .log files under log.dirs are written as plaintext bytes. There are two standard ways to get encryption at rest. First, volume/disk encryption beneath Kafka: Linux LUKS/dm-crypt on the data partition, or a cloud-managed encrypted volume such as an AWS EBS volume backed by KMS, GCP persistent disk with CMEK, or Azure disk encryption. This is transparent to Kafka and protects against stolen disks but not against a compromised broker process. Second, client-side (application-level) payload encryption: the producer encrypts the message value before send() and the consumer decrypts after poll(), so the broker only ever stores ciphertext. This protects PII even from broker operators but breaks compaction-by-content and server-side filtering. TLS (SSL) only protects data in transit, not at rest.
go deeper
Know the headline: Kafka does not encrypt files on disk by itself; you encrypt the disk (LUKS/cloud KMS) or encrypt the payload in the client.
Distinguish in-transit (TLS) vs at-rest, and disk-layer vs application-layer encryption with their different threat coverage.
Reason about which threats each layer covers (stolen disk vs malicious broker) and the trade-offs of client-side encryption (compaction, filtering, key mgmt).
Design a layered posture combining KMS-backed volumes + selective client-side PII encryption, and articulate why neither alone is sufficient.
## The core fact Apache Kafka stores messages in **log segments** — append-only files (named like `00000000000000000000.log`) inside the directories listed in the broker's `log.dirs` setting. Open-source Kafka writes these bytes to disk exactly as received; there is **no configuration that encrypts log segments at rest**. If someone reads the raw files (or steals the disk), they see the message data in the clear. This surprises people because Kafka has strong security features — but those cover *other* threats: - **TLS / SSL** (`security.protocol=SSL` or `SASL_SSL`) encrypts data **in transit** between clients and brokers, and between brokers. It does nothing for data sitting on disk. - **SASL** handles **authentication** (proving who you are). **ACLs** handle **authorization** (what you may do). Neither encrypts stored bytes. ## How teams actually achieve encryption at rest There are two layers, often combined. ### 1. Volume / disk encryption (below Kafka) Encrypt the storage that holds `log.dirs`, transparently to Kafka: - **LUKS / dm-crypt** on Linux: the block device is encrypted; the OS decrypts on read using a key held in memory. - **Cloud KMS-backed volumes**: AWS **EBS encryption** (keys in AWS KMS), GCP persistent disks with **CMEK** (customer-managed encryption keys), Azure disk encryption. The cloud provider encrypts/decrypts blocks; the key lives in a Key Management Service. **Protects against:** a stolen/decommissioned physical disk, or a snapshot leaking. **Does NOT protect against:** anyone who can read through the running broker (the data is decrypted in memory and via the filesystem), or a broker operator with shell access. ### 2. Client-side / application-level payload encryption (above Kafka) The **producer encrypts the message value before sending**; the **consumer decrypts after receiving**. The broker only ever stores ciphertext. - Implemented via a custom `Serializer`/`Deserializer`, an interceptor, or a library (e.g., envelope encryption with a KMS-issued data key). **Protects against:** even the broker operator and anyone reading the disk — true end-to-end for the payload. **Trade-offs:** key management complexity; you usually leave the **key/headers in cleartext** so partitioning still works; **log compaction by content and server-side filtering can't inspect ciphertext**; CPU cost. ## Putting it together A common production posture is **both**: KMS-backed encrypted volumes for defense-in-depth against physical/snapshot theft, **plus** client-side encryption for sensitive fields (PII) so the broker never sees plaintext. TLS is added on top for in-transit protection. The interview trap is to say 'enable Kafka's at-rest encryption setting' — there is none.
- What threat does disk/volume encryption NOT protect against?A compromised or malicious broker process/operator — the data is decrypted in memory and through the filesystem while the broker runs, so anyone reading through the live broker sees plaintext. It only protects against stolen/decommissioned disks or leaked snapshots.
- Why doesn't TLS count as encryption at rest?TLS (security.protocol=SSL/SASL_SSL) encrypts bytes only while they travel over the network. Once a broker receives and writes a message, it is decrypted and stored as plaintext in the log segment. At-rest means while stored on disk.
saying these in an interview costs you the question
- Claiming Kafka has a built-in setting to encrypt log segments at rest
- Saying TLS/SSL provides encryption at rest
- Believing disk encryption protects against a compromised broker process
- Confusing authentication (SASL) or authorization (ACLs) with encryption