How does Kafka protect sensitive dynamic broker configs (like SSL keystore passwords), and what is the role of the password encoder secret?
answer
- password.encoder.secret = static, per-broker
- sensitive dynamic configs encrypted in metadata
- --describe redacts secrets (sensitive=true)
- password.encoder.old.secret for rotation
- lose the secret = can't decrypt configs
basics
~10 sSensitive dynamic configs (e.g. passwords) are stored encrypted in the metadata store, not in plaintext. Each broker needs password.encoder.secret in server.properties to encrypt/decrypt them, and kafka-configs --describe shows such values as redacted/[hidden].
solid answer
~50 sWhen you set a sensitive dynamic broker config — keystore/truststore passwords, key passwords — via kafka-configs, Kafka encrypts it before persisting it in metadata so it is never stored in clear text. The encryption key is derived from password.encoder.secret, a static value each broker keeps in its own server.properties (you also tune password.encoder.iterations, .keyfactory.algorithm, .cipher.algorithm). Because the secret is static and per-broker, sensitive dynamic configs are normally set per-broker. If you rotate the secret you set password.encoder.old.secret so existing values can be re-encrypted. On --describe, Kafka returns sensitive values as null/redacted rather than revealing them. Crucially, the encoder secret itself must be present in server.properties before the broker can decode its persisted sensitive configs on startup; losing it means the broker can't read those values. This is distinct from non-sensitive dynamic configs, which are stored as-is.
go deeper
Know that secret configs like keystore passwords are stored encrypted, not in plaintext.
Identify password.encoder.secret as the static key and that --describe redacts secrets.
Explain per-broker encryption, the startup decryption dependency, and secret rotation via old.secret.
Define secret-management and rotation policy across the fleet and the failure modes of a lost/mismatched encoder secret.
## Why encryption is needed Dynamic broker configs are persisted in the cluster metadata store (ZooKeeper historically, the KRaft metadata log today). Some of these configs are **secrets** — TLS keystore passwords, truststore passwords, key passwords, SASL credentials. Storing them in clear text in metadata would expose them to anyone who can read that store. So Kafka **encrypts** sensitive dynamic configs at rest. ## The password encoder Kafka encrypts/decrypts these values using a **password encoder** keyed off a static secret each broker holds: ```properties # server.properties on each broker password.encoder.secret=<a-strong-cluster-operator-chosen-secret> password.encoder.iterations=4096 password.encoder.keyfactory.algorithm=PBKDF2WithHmacSHA512 password.encoder.cipher.algorithm=AES/CBC/PKCS5Padding password.encoder.key.length=256 ``` - `password.encoder.secret` — the master secret used to derive the encryption key. It is **static** (lives in `server.properties`) precisely so the broker can decrypt persisted secrets at startup, before any dynamic config is available. - The other properties tune the KDF and cipher. Because the secret is per-broker and static, **sensitive dynamic configs are typically applied per-broker** (`--entity-name <id>`), each broker encrypting with its own secret. ## Setting a sensitive config ``` kafka-configs.sh --bootstrap-server b:9092 \ --entity-type brokers --entity-name 1 \ --alter --add-config \ 'listener.name.internal.ssl.keystore.password=secretpw' ``` Kafka recognizes `*.password`/sensitive keys, encrypts the value, and stores the ciphertext. ## Reading it back ``` kafka-configs.sh ... --entity-type brokers --entity-name 1 --describe ``` Sensitive values are **not** printed in clear text — they come back as `null`/redacted (`sensitive=true`). This prevents accidental disclosure via the CLI/AdminClient. ## Rotating the secret To change `password.encoder.secret` without breaking existing encrypted values, set the previous secret as `password.encoder.old.secret` in `server.properties` on restart. The broker decrypts existing values with the old secret and re-encrypts them with the new one. After the rotation rolls through the cluster you can remove the old secret. ## Critical operational caveats - **Don't lose the secret.** If `password.encoder.secret` is missing or wrong, the broker cannot decrypt its persisted sensitive configs and will fail to apply them. - The encoder secret is itself a static config — it is never stored dynamically (chicken-and-egg: you'd need it to decrypt itself). - Non-sensitive dynamic configs are stored as plain values; only sensitive ones are encrypted. - This is unrelated to wire/at-rest data encryption — it only protects config secrets in the metadata store.
- Why must password.encoder.secret live in server.properties rather than being set dynamically?It is the key used to decrypt persisted sensitive dynamic configs at startup; it must be available statically before any dynamic config can be read — otherwise there is a chicken-and-egg decryption problem.
- How do you rotate the encoder secret without breaking existing encrypted values?Set the previous secret as password.encoder.old.secret and the new one as password.encoder.secret; on restart the broker decrypts with the old and re-encrypts with the new, then you remove the old secret.
saying these in an interview costs you the question
- Saying sensitive dynamic configs are stored in plaintext in ZooKeeper/metadata.
- Setting password.encoder.secret dynamically — it must be static.
- Expecting --describe to print passwords in clear text.
- Confusing config-secret encryption with TLS/data-at-rest encryption.