skip to content

How do you control which topics and consumer groups MM2 replicates, using allow/deny filters?

level: middleimportance: must knowfreq 60%

answer

  1. topics + topics.exclude / groups + groups.exclude
  2. Deny always beats allow
  3. Defaults exclude __.*, internal, console-consumer, connect-*
  4. refresh.topics.interval.seconds = dynamic discovery (default 600s)
  5. Pluggable topic.filter.class / group.filter.class

basics

~10 s

MM2 uses topics/topics.exclude and groups/groups.exclude regex lists (per flow). The 'topics'/'groups' allowlist says what to replicate; the 'exclude' denylist removes matches. By default internal and MM2 bookkeeping topics/groups are excluded.

solid answer

~30 s

Per source->target flow, MM2 selects topics with the `topics` allowlist (regex, default `.*`) minus the `topics.exclude` denylist (default excludes MM2 internal topics and `.*[\-\.]internal`, `.*\.replica`, `__.*`). Consumer groups are filtered the same way with `groups` (default `.*`) and `groups.exclude` (default `console-consumer-.*`, `connect-.*`, `__.*`). Deny always wins over allow. These map to the DefaultTopicFilter/DefaultGroupFilter classes, configurable via `topic.filter.class`/`group.filter.class`. The source connector refreshes its topic list every `refresh.topics.interval.seconds` so newly-created matching topics start replicating automatically. A common gotcha: MM2 by default does NOT re-replicate already-mirrored remote topics back (to prevent cycles), and it excludes its own internal topics so heartbeats/checkpoints don't loop.

go deeper

for a junior

Know that 'topics' allows and 'topics.exclude' denies, and the same pattern exists for groups.

for a middle

Know the default exclude patterns, that deny wins, and that discovery is periodic via refresh intervals.

for a senior

Discuss pluggable filter classes, cycle prevention in bidirectional setups, and tuning refresh intervals vs source-cluster load.

for a principal

Design filter conventions and naming policies across a multi-cluster mesh that avoid replication loops and accidental internal-topic capture at scale.

MM2 must decide **which topics' data** and **which consumer groups' offsets** to replicate for each directional flow. It does this with two pairs of filters, each an allow/deny list expressed as a comma-separated list of regular expressions (or literal names). **Topic selection (used by MirrorSourceConnector):** - `topics` — the **allowlist**. Default `.*` (everything). A topic must match at least one entry here to be considered. - `topics.exclude` — the **denylist** (older alias: `topics.blacklist`). Default excludes MM2's own plumbing and remote/internal topics, roughly: `.*[\-\.]internal, .*\.replica, __.*`. Anything matching here is dropped even if it matched the allowlist. **Deny wins.** **Group selection (used by MirrorCheckpointConnector):** - `groups` — allowlist, default `.*`. - `groups.exclude` — denylist (older alias `groups.blacklist`), default `console-consumer-.*, connect-.*, __.*` so throwaway console consumers and Connect's own groups aren't translated. **How they combine:** a name is replicated iff it matches the allowlist AND does not match the denylist. The denylist is evaluated last, so it always overrides. **Implementation:** these are read by pluggable classes `DefaultTopicFilter` and `DefaultGroupFilter`, swappable via `topic.filter.class` and `group.filter.class` if you need custom logic. There is also a `config.property.filter.class` (DefaultConfigPropertyFilter) controlling which **topic config keys** get synced — separate from topic selection. **Dynamic discovery:** MirrorSourceConnector periodically re-evaluates the filters against the live source-cluster topic list every `refresh.topics.interval.seconds` (default 600s/10min) and re-checks groups every `refresh.groups.interval.seconds`. So creating a new topic that matches `topics` causes replication to begin without restarting MM2 — but with up to a 10-minute delay unless you lower the interval. **Cycle prevention edge case:** in bidirectional or hub-and-spoke topologies, a topic mirrored as `A.orders` onto cluster B must not be mirrored back to A as `B.A.orders`. The default `topics.exclude` patterns (`.*\.replica`, and the remote-topic naming convention) plus MM2's awareness of already-remote topics prevent infinite replication loops. Misconfiguring the exclude list (e.g. clearing it to `''`) can create replication storms. **Gotcha:** setting `topics` to a broad regex pulls in `__consumer_offsets`-style or compacted system topics if you also weaken the default exclude — almost never what you want. Keep the default `__.*` and internal exclusions, and add specific allows.

  • You add a new topic matching the allowlist but it doesn't replicate for several minutes. Why?
    MirrorSourceConnector only re-scans the source topic list every refresh.topics.interval.seconds (default 600s = 10 min). Lower that interval for faster pickup, or restart the connector to force an immediate refresh.
  • If a topic matches both 'topics' and 'topics.exclude', is it replicated?
    No. The denylist is evaluated last and always wins, so a topic matching the exclude pattern is dropped even if it also matched the allowlist.

saying these in an interview costs you the question

  • Saying allow wins over deny — it's the opposite; deny always overrides.
  • Believing new matching topics replicate instantly — there's a refresh interval (default 10 min).
  • Clearing topics.exclude to replicate 'everything' — this can pull in internal topics and cause replication loops.
  • Confusing topic config filtering (config.property.filter.class) with topic selection (topic.filter.class).

context

resolving links…