Security

TLS encrypts data moving between clients and brokers (and between brokers) so it can't be read or tampered with on the wire. You set security.protocol=SSL and point clients at a truststore (ssl.truststore.location/password); brokers add an SSL listener plus a keystore.

It controls TLS hostname verification — whether the client checks that the broker's certificate actually matches the hostname it connected to. The default is https (enabled). Setting it to empty disables that check, which lets an attacker with any valid-looking cert impersonate the broker (man-in-the-middle).

JKS is a binary Java keystore file (password-protected, holds cert+key) configured via ssl.keystore.location plus ssl.keystore.type=JKS. PEM is plain text Base64 cert/key data; set ssl.keystore.type=PEM and you can pass the material inline or by file. PEM avoids the keytool step.

Use ssl.enabled.protocols to allow versions (e.g. TLSv1.2,TLSv1.3) and ssl.protocol for the default context; restrict algorithms with ssl.cipher.suites. Sane defaults: enable only TLSv1.2 and TLSv1.3, disable TLSv1.0/1.1, and rely on strong AEAD ciphers (GCM/ChaCha20).

Rotate leaf certs by updating each broker's keystore and reloading without a full restart — Kafka watches the keystore/truststore files and reloads them when the file changes (KIP-226 dynamic config / file watch). Rotate the CA by adding the new CA to every truststore first, migrating certs, then removing the old CA last.

With plain TLS, only the broker proves its identity with a certificate. With mutual TLS, the client ALSO presents its own certificate, so the broker authenticates the client. The client's certificate identity becomes its Kafka principal.

Set security.protocol=SSL, point ssl.keystore.location/password (and ssl.key.password) at the client's keystore holding its cert+private key, and ssl.truststore.location/password at the truststore holding the broker's CA. The broker side must have ssl.client.auth=required.

By default the principal is the full certificate Subject DN, e.g. User:CN=svc,OU=apps,O=Acme. ssl.principal.mapping.rules is an ordered list of RULE: regex transforms (like Kerberos auth_to_local) that rewrite the DN into a shorter principal such as User:svc.

With a CA-signed model the broker truststore only needs the one CA cert; any client cert signed by that CA is trusted automatically. With self-signed certs you'd have to add every individual client cert to every broker's truststore, which doesn't scale.

Certs expire, so you must renew client certs before expiry or clients drop off. Revocation (killing a compromised cert early) is hard: Kafka's TLS layer doesn't check CRLs/OCSP by default, so teams often rely on short-lived certs and rotating the CA/truststore instead.

SASL is Kafka's pluggable authentication framework. PLAIN sends a username and password (must run over TLS). SCRAM uses a salted challenge-response so the raw password never travels the wire.

Use kafka-configs.sh --alter --add-config 'SCRAM-SHA-256=[password=secret]' --entity-type users --entity-name alice. Kafka stores a salted, iterated hash (not the password). Delete with --delete-config; list with --describe.

Set sasl.jaas.config inline. PLAIN uses PlainLoginModule, SCRAM uses ScramLoginModule, each with username/password. On the broker, the KafkaServer JAAS section defines accepted users (PLAIN) or the inter-broker login.

In SCRAM the client and server exchange nonces; the client derives a salted PBKDF2 key and sends a proof, never the password. The broker stores only StoredKey/ServerKey, so even a server breach or absent TLS doesn't directly leak the password.

Use SCRAM for self-managed username/password auth with live user management. Use PLAIN only when an external system (e.g. LDAP via a custom callback) validates credentials. Prefer OAUTHBEARER/mTLS for identity-provider or certificate-based auth. Always wrap in TLS.

SASL/GSSAPI is Kafka's way of authenticating with Kerberos. Clients get a ticket from a Kerberos KDC and present it to the broker, so both sides prove who they are without sending passwords over the wire.

Set security.protocol=SASL_SSL, sasl.mechanism=GSSAPI, and sasl.kerberos.service.name=kafka. Provide a JAAS config (via sasl.jaas.config) pointing useKeyTab to a keytab file and the principal that keytab contains.

krb5.conf tells the JVM the default realm, where the KDCs are, and how DNS domains map to realms. Kerberos also requires synchronized clocks (within ~5 minutes) and consistent forward/reverse DNS, or authentication fails.

Kafka runs a background login thread that re-logs in or renews the Kerberos ticket before it expires. sasl.kerberos.ticket.renew.window.factor (default 0.8) sets how far into the ticket's lifetime to wait before renewing — at 80% of its life.

Point krb5.conf at the AD domain controllers (AD is a KDC), create service accounts and SPNs in AD, export keytabs, and use sasl.kerberos.principal.to.local.rules (or a custom KafkaPrincipalBuilder) to turn full principals like [email protected] into Kafka users like alice for ACL matching.

OAUTHBEARER is a Kafka SASL mechanism where the client presents an OAuth2 bearer token (usually a JWT) instead of a username/password. Kafka validates the token to authenticate the connection.

The broker fetches the IdP's public keys from a JWKS endpoint and checks the JWT's signature, issuer, audience, and expiry. Configs like sasl.oauthbearer.jwks.endpoint.url, expected.issuer, expected.audience, and sub.claim.name control this.

The client's login manager proactively re-acquires a new token before the old one expires, based on a fraction of the token lifetime. Established connections stay open after a token expires; expiry mainly gates new connections (re-authentication can enforce expiry on live ones).

Delegation tokens are lightweight shared-secret credentials a client first authenticates (e.g. via Kerberos or OAUTHBEARER), then requests from the broker. Workers/tasks use the token to authenticate to Kafka without distributing the original credential, simplifying secret distribution in distributed jobs.

Use OAUTHBEARER when each job/worker can talk to your IdP for its own short-lived token (best for centralized identity and revocation). Use delegation tokens when a coordinator authenticates once and must fan out cheap, revocable credentials to many executors without contacting the IdP per task.

An ACL (access control list entry) is a rule saying a principal (user) is allowed or denied a specific operation (like Read or Write) on a resource (like a topic), optionally from a specific host.

A producer needs Write on the topic. A consumer needs Read on the topic plus Read on its consumer group. Both implicitly get Describe from Write/Read.

Kafka is deny-by-default: if nothing matches, access is denied. If any matching DENY exists, access is denied even if an ALLOW also matches. An ALLOW grants only when no DENY matches.

A wildcard ACL uses the literal name '*' to match all resources of a type. A prefixed ACL matches every resource whose name starts with a given prefix. You select prefixed mode with --resource-pattern-type prefixed.

On KRaft set authorizer.class.name to StandardAuthorizer; on legacy ZooKeeper clusters use AclAuthorizer (formerly SimpleAclAuthorizer). KRaft stores ACLs in the metadata log; ZooKeeper-based authorizers store them in ZooKeeper.

super.users is a broker config listing principals that are granted full access and bypass all ACL checks. Any principal in the list is always authorized for every operation, no ACLs needed.

It controls the fallback when no ACL matches a resource: if true, access is allowed; if false, access is denied. The default is false (default-deny), which is the recommended secure posture.

Order: super user → ALLOWED. Else any matching DENY → DENIED. Else any matching ALLOW → ALLOWED. Else fall back to allow.everyone.if.no.acl.found (true=allow, false=deny). DENY always beats ALLOW.

authorizer.class.name selects the pluggable Authorizer the broker uses. If unset, no authorization runs (allow all). KRaft clusters use StandardAuthorizer; legacy ZooKeeper clusters used AclAuthorizer.

Kafka turns an authenticated connection into a KafkaPrincipal (type:name) via a KafkaPrincipalBuilder. By default mTLS uses the full certificate DN and SASL uses the username. A custom builder lets you remap that to cleaner or group-based principal names.

Kafka has four security protocols: PLAINTEXT (no encryption, no auth), SSL (TLS encryption, optional client cert auth), SASL_PLAINTEXT (SASL auth, no encryption), and SASL_SSL (SASL auth plus TLS encryption).

listeners is where the broker binds and accepts connections. advertised.listeners is the address the broker hands back to clients in metadata so they can reconnect. If the advertised address is unreachable, clients connect once for metadata, then fail to reach the broker.

listener.security.protocol.map maps each named listener to one of the four security protocols. With custom listener names you must define this map; it lets one broker expose, for example, an SSL internal listener and a SASL_SSL external listener at the same time.

inter.broker.listener.name selects which named listener brokers use to talk to each other (replication, controller coordination). It is mutually exclusive with security.inter.broker.protocol — you set one or the other. It lets broker-to-broker traffic run on a dedicated, often more-trusted listener separate from client traffic.

Define two named listeners: INTERNAL (private network) and EXTERNAL (internet). Map INTERNAL to SSL/mTLS and EXTERNAL to SASL_SSL via listener.security.protocol.map, advertise each with its reachable address, and point inter.broker.listener.name at INTERNAL.

Quotas are per-client limits that Kafka brokers enforce to cap how much produce/fetch traffic (bytes per second) or broker request time a client can use, stopping one noisy client from starving others.

Use kafka-configs.sh --alter --add-config 'producer_byte_rate=...' against --entity-type users --entity-name <user>. A specific entity quota overrides the cluster --entity-default for that level.

Byte-rate quotas only limit data volume, so a client sending many tiny but expensive requests can saturate broker CPU without moving many bytes. request_percentage caps the share of broker IO/network thread time a client uses, covering that gap.

The broker tracks each client's rate over a sliding window of quota.window.num samples, each quota.window.size.seconds long. When the average exceeds the quota, it computes a delay = how long the client must pause to bring the windowed average back under the limit, then delays the response by that much.

Controller mutation quotas limit how fast a user/client-id can create or delete topics and partitions, protecting the controller from a flood of metadata changes. They throttle metadata-mutation rate, whereas request_percentage throttles general request CPU on data-path brokers.

No. Open-source Kafka has no built-in encryption of log segment files on disk. Encryption at rest is provided externally: encrypt the broker's volume/disk (LUKS or a cloud KMS-backed encrypted EBS volume), or encrypt the message payload before producing it.

Encrypt the sensitive payload in the producer before send() and decrypt in the consumer after poll(), typically via a custom Serializer or interceptor using envelope encryption: a KMS issues a data key that encrypts the message, and the wrapped data key travels alongside. The broker stores only ciphertext.

Kafka's authorizer logs allowed/denied authorization decisions. Enable the authorizer logger (kafka.authorizer.logger) at INFO/DEBUG to capture which principal was allowed or denied which operation on which resource. Ship those logs to a central, tamper-evident store. Enterprise distributions (Confluent) add structured audit-log topics.

On a compacted topic, produce a tombstone — a record with the user's key and a null value. Log compaction eventually removes all prior records for that key and then drops the tombstone after delete.retention.ms. Deletion is asynchronous, not immediate, and only works per key.

Volume encryption (KMS-backed EBS, LUKS) protects against stolen disks and leaked snapshots but the running broker still sees plaintext. Client-side payload encryption protects PII even from the broker/operator but breaks content-based features and adds key-management cost. Mandate both for regulated PII: defense-in-depth plus operator-proof confidentiality.

In ZooKeeper-based Kafka, ZooKeeper stores critical metadata: topic configs, ACLs, SCRAM credentials, and broker registrations. An unauthenticated client can read or delete this metadata directly, bypassing Kafka entirely, so ZooKeeper must require authentication.

KRaft controllers form a Raft quorum that replicates the metadata log. You secure their listener (named in controller.listener.names) with its own SSL/SASL entry in listener.security.protocol.map, separate from client listeners, because that channel carries all cluster metadata.

SASL is set up via a JAAS Client section so brokers authenticate to ZooKeeper. zookeeper.set.acl=true makes Kafka write znodes with restrictive ACLs. zookeeper.ssl.client.enable=true plus keystore/truststore turns on TLS encryption to ZooKeeper.

In ZK-mode, SCRAM hashes and ACLs live in znodes protected by ZooKeeper ACLs (zookeeper.set.acl) plus SASL/TLS. In KRaft they live in the __cluster_metadata log, protected by securing the controller listener. KRaft's bootstrap challenge: seed the first SCRAM credential with kafka-storage --add-scram.

During ZK→KRaft migration (KIP-866) both systems run together and metadata is dual-written, so you must keep ZooKeeper hardened (SASL/TLS/ACLs) AND secure the new controller listener at the same time. The migration controllers still connect to ZK, so neither attack surface can be relaxed until ZK is removed.

Kafka's authorizer writes an audit trail via a dedicated logger named kafka.authorizer.logger. Set that log4j logger to DEBUG to see allowed requests too; at INFO you only see denials. Route it to its own file appender.

Check two different sources. If failed-authentication-total is climbing on that listener, it's an authentication problem (bad cert/credential). If the authorizer log shows a 'Denied' Write line for that principal and topic, it's an ACL/authorization problem. They're surfaced separately.

The broker exposes per-listener authentication counters under kafka.server:type=socket-server-metrics, including failed-authentication-total / failed-authentication-rate and successful-authentication-total. Scrape them via JMX (e.g. JMX exporter to Prometheus) and alert when the failed rate spikes.

expired-connections-killed-count counts connections the broker closed because a SASL credential (e.g. an OAuth token or Kerberos ticket) expired and the client did not re-authenticate in time. It is driven by connections.max.reauth.ms (KIP-368).

Tap two sources: the authorizer audit log (kafka.authorizer.logger, allow/deny lines) and the broker authentication metrics (failed-authentication, expired-connections-killed). Ship logs via a forwarder and metrics via a JMX/Prometheus exporter into the SIEM, normalizing principal, resource, listener, and result.