skip to content

Which Kafka ACLs does MirrorMaker 2's principal need on the source and target clusters for topic + offset replication to work?

level: middleimportance: must knowfreq 55%

answer

  1. Source: Read topic+group, Describe
  2. Target: Write + Create + Alter/DescribeConfigs
  3. 3 connectors: Source/Checkpoint/Heartbeat
  4. Connect internal topics need their own ACLs
  5. Errors = TopicAuthz / GroupAuthz exceptions

basics

~20 s

On the source: Read on the mirrored topics and the consumer group, plus Describe. On the target: Write and Create on the mirrored/internal topics, plus DescribeConfigs/AlterConfigs for config sync. MM2 also needs access to its Connect internal topics on whichever cluster hosts them.

solid answer

~40 s

MM2 runs three connector types and each needs different ACLs. The MirrorSourceConnector consumes the source, so the MM2 principal needs `Read` on the source topics and on its consumer group (Topic + Group resources) plus `Describe`/`DescribeConfigs` to discover topics and configs. It produces to the target, so it needs `Write` and `Create` on the target topics (Create because MM2 auto-creates `source.<topic>`), plus `DescribeConfigs`/`AlterConfigs` so the config-sync feature can mirror topic settings. The MirrorCheckpointConnector reads source consumer-group offsets (`Describe`/`Read` on Group) and writes checkpoints to the target. The MirrorHeartbeatConnector writes the heartbeats topic on the target. Separately, the underlying Kafka Connect runtime needs Read/Write/Create on its internal offsets/config/status topics and its Connect group on whichever cluster hosts them. Missing any of these surfaces as `TopicAuthorizationException` or `GroupAuthorizationException`, not a connectivity error.

go deeper

for a junior

Know MM2 needs Read on the source and Write on the target, granted via ACLs to its principal.

for a middle

Enumerate the per-cluster ACLs (Read/Describe/Group on source; Write/Create/Alter on target) and the three connectors.

for a senior

Include Connect internal-topic ACLs, config-sync AlterConfigs, and diagnose authz exceptions; scope with PREFIXED patterns.

for a principal

Define least-privilege ACL templates and automate their provisioning so every new replication flow is reproducibly authorized.

## What an ACL is A Kafka **ACL** (Access Control List entry) is a rule: *principal P is Allowed/Denied operation O on resource R from host H*. Resources have a **type** (`Topic`, `Group`, `Cluster`, `TransactionalId`, `DelegationToken`) and a **pattern** (`LITERAL` exact name or `PREFIXED`). Operations include `Read`, `Write`, `Create`, `Describe`, `DescribeConfigs`, `AlterConfigs`, `IdempotentWrite`, `ClusterAction`, etc. With the standard `StandardAuthorizer` (KRaft) / `AclAuthorizer`, everything is denied unless an Allow ACL matches. MM2 is just clients, so it needs the same ACLs any consumer/producer/admin would — but on *two* clusters, and for *three* connectors. ## MM2's connectors and what each does 1. **MirrorSourceConnector** — consumes records from the source cluster, produces them to the target as `source-alias.<topic>`. 2. **MirrorCheckpointConnector** — translates source consumer-group offsets into target offsets so consumers can fail over; writes a `<source>.checkpoints.internal` topic on the target. 3. **MirrorHeartbeatConnector** — periodically writes a `heartbeats` topic to verify the link is live. ## Source cluster ACLs (MM2 reads here) - `Read` on the mirrored **Topics** (LITERAL or PREFIXED to cover many). - `Describe` / `DescribeConfigs` on those topics (topic discovery + config mirroring source). - `Read` + `Describe` on the **Group** MM2 uses to consume. - For offset sync, `Describe` on the source consumer **Groups** whose offsets are checkpointed. ## Target cluster ACLs (MM2 writes here) - `Write` on the target topics. - `Create` on **Topic** (often PREFIXED) so MM2 can auto-create `source.<topic>`, plus the internal `checkpoints`/`heartbeats`/`offset-syncs` topics. - `DescribeConfigs` and `AlterConfigs` on target topics for the **config-sync** feature (mirrors partition counts, retention, etc.). - If the producer is idempotent/transactional: `IdempotentWrite` on the `Cluster` (older brokers) and possibly `Write`/`Describe` on a `TransactionalId`. ## Connect runtime ACLs (often overlooked) The Kafka Connect worker hosting MM2 keeps its own state in three internal topics — `offset`, `config`, `status` — and uses a Connect **group** for coordination. The worker's principal needs `Read`/`Write`/`Create` on those topics and `Read` on that group, on whichever cluster you configured as the Connect backing store. Forgetting these means MM2 won't even start its rebalance, which looks like a hang rather than an auth error on the data path. ## Failure signatures - Missing source `Read` → `TopicAuthorizationException` for the source topic. - Missing target `Write`/`Create` → `TopicAuthorizationException` on the target / topic-creation failure. - Missing `Group` ACL → `GroupAuthorizationException`. - Missing `AlterConfigs` → config sync silently no-ops or logs authorization warnings while data still flows. ## Edge cases - **PREFIXED ACLs** keep the rule count manageable when mirroring many topics, but a too-broad prefix grants more than intended — scope to the replication prefix. - Auto-creation requires either broker `auto.create.topics.enable` or MM2's own `Create` ACL + `replication.policy` settings; relying on broker auto-create across a secured boundary is fragile. - With a **custom replication policy** (e.g. `IdentityReplicationPolicy`, no `source.` prefix), Create/Write ACL patterns must match the *unprefixed* names — a common mismatch.

  • Why does MM2 need Create on the target, not just Write?
    MM2 auto-creates the renamed/internal topics (source.<topic>, checkpoints, heartbeats, offset-syncs). Without Create it can't materialize them, so even with Write the first replication of a new topic fails.
  • MM2 connects fine and authenticates but no data moves — where do you look first?
    Authorization. Check broker authorizer logs for TopicAuthorizationException/GroupAuthorizationException; the principal is authenticated but missing Read on source or Write/Create on target, or the Connect internal-topic ACLs are missing so the worker never rebalances.

saying these in an interview costs you the question

  • Saying Write alone is enough on the target (Create is needed for auto-created topics)
  • Forgetting the Connect internal topic + group ACLs and blaming connectivity
  • Assuming authentication implies authorization
  • Granting cluster-wide Read/Write 'to be safe' instead of scoped/PREFIXED ACLs

context

resolving links…