skip to content

A client suddenly can't produce to a topic. How do you tell from monitoring whether it's an authentication failure or an authorization (ACL) denial?

level: middleimportance: must knowfreq 55%

answer

  1. authn = socket-server-metrics; authz = authorizer.logger
  2. handshake/SASL error vs TopicAuthorizationException
  3. connection fails vs connection OK + op refused
  4. kafka-acls.sh --list to confirm
  5. txn/idempotent producers need cluster/txn-id ACLs

basics

~20 s

Check two different sources. If failed-authentication-total is climbing on that listener, it's an authentication problem (bad cert/credential). If the authorizer log shows a 'Denied' Write line for that principal and topic, it's an ACL/authorization problem. They're surfaced separately.

solid answer

~40 s

Authentication and authorization fail in different layers and are observed in different places. Authentication (TLS mutual auth or SASL login) happens first; if it fails, the broker increments failed-authentication-total under kafka.server:type=socket-server-metrics for that listener and the connection never establishes — the client sees handshake/auth errors, not an authorization error. Authorization (ACL check) happens after a successful connection, per request; a denial is written to kafka.authorizer.logger as a 'Denied' line (INFO level) naming the principal, Write operation, and the topic, and the client receives TopicAuthorizationException / not-authorized. So the diagnostic is: rising failed-authentication metric with no successful connection => authentication; successful connection but Denied lines in the authorizer log and an authorization exception client-side => ACL. Confirm ACL state with kafka-acls.sh --list. This separation is exactly why a complete audit pipeline taps both sources.

go deeper

for a junior

Know to check failed-authentication metrics for login problems and the authorizer log for Denied lines.

for a middle

Map client exception types (SaslAuthenticationException vs TopicAuthorizationException) to the right source and confirm with kafka-acls.sh.

for a senior

Handle subtle cases like transactional/idempotent producer ACLs and rule out non-security causes.

for a principal

Codify this triage into runbooks and dashboards so on-call can disambiguate authn vs authz in seconds.

## Two layers, two failure modes When a Kafka client interacts with a broker, security is checked in two ordered stages: 1. **Authentication** — the connection-establishment step. With TLS this is the (mutual) TLS handshake; with SASL it's the login exchange (SCRAM/PLAIN/GSSAPI/OAUTHBEARER). It proves *identity*. 2. **Authorization** — performed *per request* after the connection is up. The authorizer checks ACLs to decide if this principal may perform this operation (e.g. Write) on this resource (the topic). It proves *permission*. A produce failure can come from either stage, and they look different in monitoring. ## Symptom A — Authentication failure - **Where it shows:** `kafka.server:type=socket-server-metrics` → `failed-authentication-total/-rate` increments on the affected **listener**. The connection does **not** establish. - **Client-side:** SSL handshake exceptions, `SaslAuthenticationException`, or generic connection failures — *not* an authorization error. - **Typical causes:** expired/untrusted client certificate, wrong SCRAM password, expired Kerberos ticket, invalid OAuth token, wrong security.protocol. - **Not in the authorizer log**, because the request never reached authorization. ## Symptom B — Authorization (ACL) denial - **Where it shows:** `kafka.authorizer.logger` logs a **Denied** line at INFO: principal `User:svc-orders`, operation `Write`, resource `Topic:LITERAL:payments`, host, result `Denied`. - **Client-side:** `TopicAuthorizationException` / 'Not authorized to access topics' — the connection succeeded, but the operation was refused. - **Typical causes:** missing Write ACL on the topic (or on the transactional id / cluster for idempotent/transactional producers), wrong principal mapping, ACLs not yet propagated. - **Confirm with:** `kafka-acls.sh --bootstrap-server ... --list --topic payments`. ## The decision procedure 1. Look at `failed-authentication-rate` for the client's listener. Climbing? => authentication; fix the credential/cert. Flat? => the client authenticated fine; go to step 2. 2. Grep the authorizer log for the principal and topic. A `Denied` Write line => authorization; add/fix the ACL. 3. Cross-check the **client exception type**: handshake/SASL error => authn; TopicAuthorizationException => authz. ## Edge cases / gotchas - **Idempotent/transactional producers** need more than topic Write — they need Write on the *cluster* (IdempotentWrite, or in newer versions auto-granted with topic Write) and Write/Describe on the *transactional id*. A denial here can be confusing: the client *can* connect and even describe, but the produce path is denied — clearly an authz issue in the authorizer log. - A **flat** failed-auth metric plus **no** Denied line usually points to something non-security (e.g. NOT_LEADER_OR_FOLLOWER, topic doesn't exist, min.insync.replicas) — don't force a security diagnosis. - Authorizer denials are at INFO by default, so you don't need DEBUG to see *denials* — only to see *allows*.

  • What client-side exception distinguishes an ACL denial from an auth failure for a producer?
    An ACL denial surfaces as TopicAuthorizationException ('Not authorized to access topics'), meaning the connection succeeded but the operation was refused. An authentication failure surfaces as SaslAuthenticationException or a TLS handshake error, meaning the connection never authenticated.
  • An idempotent producer authenticates fine and can describe the topic but produce is denied — where do you look and why?
    Look in kafka.authorizer.logger for Denied lines: idempotent/transactional producers need Write on the cluster (IdempotentWrite) and on the transactional id, beyond topic Write. The denial is an authorization issue even though authentication and describe succeed.

saying these in an interview costs you the question

  • Expecting an ACL denial to appear in failed-authentication metrics.
  • Expecting an auth/handshake failure to show up in the authorizer log.
  • Forcing a security explanation when both signals are clean (could be NOT_LEADER, missing topic, min.insync.replicas).

context

resolving links…