How do you configure SASL and TLS between Kafka brokers and ZooKeeper, and what do zookeeper.set.acl and zookeeper.ssl.client.enable control?

## The two independent concerns Securing the broker↔ZooKeeper link has two orthogonal layers: - **Authentication (SASL)** — proving *who* the broker is, and using that identity to restrict znode access. - **Encryption (TLS)** — protecting the *channel* so metadata and credentials can't be sniffed. You can enable either independently, but production hardening uses both. ## SASL authentication ZooKeeper supports SASL via Java's JAAS framework. You supply a JAAS login config (commonly `-Djava.security.auth.login.config=/path/jaas.conf`). On the **broker** side the file has a `Client` section — for DIGEST-MD5: ``` Client { org.apache.zookeeper.server.auth.DigestLoginModule required username="kafka" password="secret"; }; ``` or a Kerberos `Client` using `com.sun.security.auth.module.Krb5LoginModule`. On the **ZooKeeper server** side you set a `Server` JAAS section with matching credentials and enable the SASL authentication provider (`authProvider.sasl=org.apache.zookeeper.server.auth.SASLAuthenticationProvider`) and, for DIGEST, the digest provider. Now ZooKeeper knows the authenticated identity of each connecting broker. ## zookeeper.set.acl Set in the **Kafka broker** config: ``` zookeeper.set.acl=true ``` When true, Kafka creates its znodes with ZooKeeper ACLs that grant **write** access only to the authenticated Kafka principal (and typically world read for non-sensitive nodes, with sensitive nodes locked tighter). Without it, znodes are created world-writable. **Crucially**, this only affects znodes Kafka creates *after* the flag is enabled — it does not retroactively secure existing nodes. ## TLS to ZooKeeper TLS requires **ZooKeeper 3.5+** and the **Netty** connection factory on the server (`serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory`) plus a secure client port (default 2182). On the **Kafka broker**, Kafka exposes ZK TLS settings prefixed `zookeeper.ssl.`: ``` zookeeper.ssl.client.enable=true zookeeper.ssl.keystore.location=/path/zk.keystore.jks zookeeper.ssl.keystore.password=... zookeeper.ssl.truststore.location=/path/zk.truststore.jks zookeeper.ssl.truststore.password=... ``` Kafka translates these into the underlying `zookeeper.client.secure`, `zookeeper.ssl.keyStore.*`, and `zookeeper.ssl.trustStore.*` system properties the ZK client reads. Mutual TLS (broker cert verified by ZK) can also serve as the authentication mechanism via the X509AuthenticationProvider, so TLS can do double duty as authN. ## Retrofitting an existing cluster Because `zookeeper.set.acl=true` doesn't touch existing znodes, an unsecured cluster being hardened must run: ``` bin/zookeeper-security-migration.sh --zookeeper.acl=secure --zookeeper.connect=... ``` This walks the existing znode tree and applies the secure ACLs. To reverse (e.g., decommissioning), use `--zookeeper.acl=unsecure`. The migration tool itself must run with broker credentials. ## Edge cases - Enabling TLS but forgetting the Netty factory on the ZK server side causes connection failures. - DIGEST-MD5 passwords sit in the JAAS file in plaintext — protect file permissions; many shops prefer mTLS or Kerberos. - If you enable set.acl but later lose the broker principal, you can lock yourself out of the znodes (recover via `skipACL=yes` superuser on the ZK server).