skip to content

How do you turn on an audit log of allowed and denied authorization decisions in Apache Kafka, and where does that output go?

level: juniorimportance: must knowfreq 70%

answer

  1. logger name: kafka.authorizer.logger
  2. INFO = denies, DEBUG = allows
  3. additivity=false, own appender
  4. StandardAuthorizer / AclAuthorizer
  5. principal+op+resource+host+Allowed/Denied

basics

~20 s

Kafka's authorizer writes an audit trail via a dedicated logger named kafka.authorizer.logger. Set that log4j logger to DEBUG to see allowed requests too; at INFO you only see denials. Route it to its own file appender.

solid answer

~40 s

Every authorization check in Kafka's standard authorizer (StandardAuthorizer in KRaft, or the legacy AclAuthorizer) is logged through a separate Log4j logger named kafka.authorizer.logger. Denied operations are logged at INFO level; allowed operations are logged at DEBUG. So to get a full audit trail you configure that logger to DEBUG in log4j2.yaml (or log4j.properties on older builds) and attach it to its own rolling file appender, keeping it out of the noisy main server log. Each line includes the principal, operation, resource (type/name/pattern), the requested host, and whether it was Allowed or Denied. Operationally you usually leave it at INFO in steady state (denials only, low volume) and flip to DEBUG when you need to prove what a principal is allowed to do or feed allow/deny events to a SIEM.

go deeper

for a junior

Know there is a dedicated kafka.authorizer.logger and that DEBUG captures allowed requests while INFO shows denials.

for a middle

Configure a separate appender with additivity=false and understand the line contents (principal/op/resource/host/result).

for a senior

Reason about volume trade-offs, super.users and allow.everyone.if.no.acl.found interactions, and steady-state INFO vs investigative DEBUG.

for a principal

Define org-wide audit policy: routing to SIEM, retention, sampling, and how authorizer logs fit a defense-in-depth strategy alongside auth metrics.

## What this is In Kafka, an *authorizer* is the pluggable component that decides whether a given client (a *principal*, e.g. `User:alice`) is allowed to perform an *operation* (Read, Write, Describe, Create, etc.) on a *resource* (a topic, group, cluster, transactional id, etc.). The decision is driven by *ACLs* (Access Control List entries). Kafka's built-in authorizer is `StandardAuthorizer` in modern KRaft clusters and `AclAuthorizer` (formerly `SimpleAclAuthorizer`) in ZooKeeper-based clusters. ## The audit logger Whenever the authorizer makes a decision, it emits a log line through a **separate, dedicated logger** whose name is `kafka.authorizer.logger`. This is intentional: it lets you control audit verbosity and routing independently from the broker's general logging. The key level convention: - **Denied** authorization decisions are logged at **INFO**. - **Allowed** authorization decisions are logged at **DEBUG**. So with the default INFO threshold you get a record of every *denial* (the security-interesting events) without drowning in allow noise. To capture a *complete* audit trail (every allow and deny), you raise `kafka.authorizer.logger` to **DEBUG**. ## How to configure it In modern Kafka the logging backend is Log4j2 (`log4j2.yaml`). You define a dedicated appender (e.g. a `RollingFile`) and bind the logger to it with `additivity=false` so audit lines don't also pollute the root server log: ```yaml Loggers: Logger: - name: kafka.authorizer.logger level: DEBUG additivity: false AppenderRef: - ref: authorizerAppender ``` On older 2.x/3.x builds using Log4j 1.x this is `log4j.logger.kafka.authorizer.logger=DEBUG, authorizerAppender` plus `log4j.additivity.kafka.authorizer.logger=false` in `log4j.properties`. ## What each line contains A typical denied line looks conceptually like: principal `User:alice`, operation `Write`, resource `Topic:LITERAL:payments`, host `10.0.0.5`, result `Denied`, based on ACLs. Allowed lines are the same shape with `Allowed`. ## Edge cases / gotchas - **Super users** (`super.users`) short-circuit the authorizer and are allowed; with allow-logging on you'll still see their actions logged as allowed, which is useful for audit. - `allow.everyone.if.no.acl.found=true` changes behavior (default allow when no ACL exists) — those allows only show at DEBUG, which can hide unintended broad access if you only watch INFO. - Turning on DEBUG on a busy cluster produces very high log volume — size the appender and rotation accordingly, or sample. - The logger only covers authorization, not authentication; failed logins are surfaced via metrics/other logs, not this logger.

  • At default INFO level, do you see allowed requests in the authorizer log?
    No. At INFO only denials are logged. Allowed decisions are at DEBUG, so you must raise kafka.authorizer.logger to DEBUG to capture the full allow/deny trail.
  • Why set additivity=false on the authorizer logger?
    So audit events go only to the dedicated audit appender and are not duplicated into the root/server log, keeping the audit trail clean, separately rotated, and easy to ship to a SIEM.

saying these in an interview costs you the question

  • Saying allowed and denied are both logged at INFO by default (allows are DEBUG).
  • Claiming the authorizer logger records failed authentications (it records authorization, not authentication).
  • Confusing the dedicated kafka.authorizer.logger with the general kafka root logger.

context

resolving links…