skip to content

How do you audit authentication and authorization (ACL) decisions in Kafka for governance and compliance?

level: middleimportance: should knowfreq 40%

answer

  1. kafka.authorizer.logger
  2. Denied=INFO, Allowed=DEBUG by default
  3. StandardAuthorizer (KRaft) / AclAuthorizer
  4. Ship to SIEM, append-only retention
  5. Confluent audit = CloudEvents to topics

basics

~20 s

Kafka's authorizer logs allowed/denied authorization decisions. Enable the authorizer logger (kafka.authorizer.logger) at INFO/DEBUG to capture which principal was allowed or denied which operation on which resource. Ship those logs to a central, tamper-evident store. Enterprise distributions (Confluent) add structured audit-log topics.

solid answer

~50 s

Kafka's pluggable authorizer (StandardAuthorizer in KRaft, formerly AclAuthorizer/SimpleAclAuthorizer) emits authorization decisions through the dedicated kafka.authorizer.logger. By default denied requests log at INFO and allowed requests at DEBUG; you configure log4j to route this logger to its own appender. Each entry records the principal, operation (READ/WRITE/DESCRIBE/etc.), resource (topic/group/cluster), host, and Allowed/Denied outcome — the raw material for an audit trail. For authentication, broker logs and SASL/SSL handshake logs show who connected. For real governance you forward these to a centralized, append-only, tamper-evident system (SIEM/ELK) with retention matching your compliance window, since broker-local logs rotate and aren't immutable. Confluent Platform adds structured audit logging that publishes auth/authz events to dedicated Kafka topics in a CloudEvents-style schema, which is far more queryable than parsing log4j lines. The key governance point: capture every Allowed and Denied decision, centralize it, and protect it from tampering.

go deeper

for a junior

Know that Kafka logs ACL allow/deny decisions via the authorizer logger.

for a middle

Name kafka.authorizer.logger, the default INFO(denied)/DEBUG(allowed) split, and the need to centralize logs.

for a senior

Discuss StandardAuthorizer vs AclAuthorizer, runtime log-level changes, super.user blind spots, and SIEM/tamper-evidence requirements.

for a principal

Design a compliance-grade audit pipeline (Confluent CloudEvents audit topics or SIEM), retention, alerting, and auditing of ACL changes themselves.

## Why audit logging matters Governance and regulations (SOC 2, GDPR, PCI) require you to prove **who accessed what, and which access was allowed or denied**. In Kafka that means capturing **authentication** (who connected) and **authorization** (what each principal was permitted to do). ## The authorizer and its logger Kafka authorization is handled by a pluggable **authorizer**: - **KRaft mode:** `StandardAuthorizer` (set via `authorizer.class.name`). - **Older/ZooKeeper:** `AclAuthorizer` (earlier `SimpleAclAuthorizer`). The authorizer evaluates **ACLs** (access control list rules: principal + operation + resource + permission) on every request and emits the outcome through a dedicated SLF4J/log4j logger named **`kafka.authorizer.logger`**. Defaults: - **Denied** requests log at **INFO**. - **Allowed** requests log at **DEBUG** (off by default to avoid volume). To audit *allowed* access too, raise this specific logger to **DEBUG** and route it to its own appender so it doesn't drown the main broker log. A typical line includes the **principal** (e.g., `User:alice`), **operation** (`READ`, `WRITE`, `DESCRIBE`, `CREATE`, `ALTER`…), **resource** (`Topic:payments`, `Group:...`, `Cluster`), **client host/IP**, and **Allowed/Denied**. You can change a logger level at runtime without restart via the **admin API** (`kafka-configs`/`incrementalAlterConfigs` on broker loggers) — useful to turn on verbose audit temporarily. ## Authentication auditing For **who connected**, you rely on broker logs of **SASL/SSL handshakes** and connection events: the authenticated principal, the SASL mechanism (PLAIN, SCRAM, GSSAPI/Kerberos, OAUTHBEARER) or mTLS certificate DN, and failures. There's no single 'auth audit topic' in OSS; you mine the broker/network logs. ## Making it real governance, not just logs Broker-local log4j files **rotate and are mutable** — inadequate as an audit system of record. Production practice: - **Ship** authorizer + connection logs to a **centralized SIEM** (Splunk, ELK/Elasticsearch, Loki) via a log agent. - Make the store **append-only / tamper-evident** with **retention matching the compliance window**. - Alert on patterns (spikes in Denied, access from new hosts, privileged ACL changes). ## Confluent's structured audit logging Confluent Platform/Cloud provides **audit logs** that publish **structured auth/authz events** to dedicated Kafka topics using a **CloudEvents** JSON schema (authentication events and authorization events with principal, action, resource, granted/denied, etc.). This is queryable and machine-parseable, unlike grepping log4j text, and integrates with RBAC. It's the enterprise answer to the OSS 'parse the logger' approach. ## Edge cases - **Volume:** logging every Allowed request on a busy cluster is huge; sample or scope to sensitive topics. - **ACL changes themselves** should be audited (who granted/revoked) — track CreateAcls/DeleteAcls operations. - **Authorizer caching** and **super.users** bypass ACL checks — super-user actions may not appear as ordinary ACL decisions, a known blind spot. ## What NOT to say - 'Kafka has a built-in immutable audit log out of the box' — OSS gives you a *logger*, not a managed audit system. - 'TLS logs cover authorization' — TLS is transport/auth identity, not the authz decision trail.

  • By default, why might allowed accesses not show up in your audit log even though denials do?
    Because the authorizer logs denied decisions at INFO but allowed decisions at DEBUG. Unless you raise kafka.authorizer.logger to DEBUG, successful accesses aren't recorded — only denials are visible at default levels.
  • Why aren't broker-local log4j files sufficient as an audit system of record?
    They rotate and are mutable, so entries can be lost or altered and retention is limited. Compliance requires a centralized, append-only, tamper-evident store (SIEM) with retention matching the regulatory window.

saying these in an interview costs you the question

  • Claiming Kafka OSS ships a built-in immutable audit-log system
  • Saying allowed accesses are logged by default (they're DEBUG)
  • Confusing TLS/authentication logs with the authorization decision trail
  • Forgetting super.users bypass ACL checks and may not appear as decisions
  • Leaving audit logs only on the broker disk with default rotation

context

resolving links…