Design how a multi-tenant Connect cluster gives each connector its own broker identity for per-tenant ACLs. What pieces fit together?

**The core gap:** A single Connect worker process opens its producer/consumer/admin clients with one set of credentials. Every connector it hosts therefore authenticates to the brokers as the *same* principal. On the broker, ACLs are granted to principals — so if all connectors share one principal, you cannot isolate tenant A's connector from tenant B's topics. Multi-tenant Connect needs each connector to present a *distinct* identity. **Layer 1 — allow per-connector security overrides (worker):** Set `connector.client.config.override.policy=Principal`. With the default `None`, connectors can't override anything. `Principal` permits overriding exactly the security/identity-related client configs (`security.protocol`, `sasl.*`, `ssl.*`) and nothing else — so a connector can change *who it is* but not, say, `bootstrap.servers` (which `All` would dangerously permit). This is the linchpin: it scopes the blast radius of override capability to identity only. **Layer 2 — give each connector its own credentials (connector config):** In each connector's config: ``` producer.override.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \ username="tenantA" password="${vault:secret/connect/tenantA:password}"; ``` and for sinks the `consumer.override.sasl.jaas.config` equivalent. Crucially the password is a **ConfigProvider** placeholder, not plaintext, so it never lands in the internal config topic or `GET /connectors/{name}/config`. The worker resolves it at start time on each node. **Layer 3 — broker authorization (per-principal ACLs):** On the brokers, grant ACLs (or vendor RBAC) so `User:tenantA` may only Write to tenantA's target topics (source) or Read its source topics + use its consumer group (sink), and nothing else. Now even if a tenant tampers with topic names in their connector config, the broker denies access outside its grant. Add **client quotas** per principal to prevent a noisy tenant starving others. **Layer 4 — protect the control plane (REST):** Per-tenant data isolation is moot if any tenant can read another's connector config or delete it. Secure the REST API (HTTPS + an auth extension or a gateway with RBAC) so tenants only manage their own connectors. ConfigProvider externalization also ensures a tenant reading a config they *can* see doesn't get another's secret. **The worker's own identity:** Keep the worker's internal config/offset/status and group-membership clients on a dedicated *operator* principal set at the **top level** of the worker config (not the producer.override./consumer.override. prefixes, which only affect connector clients). That principal needs ACLs on the internal Connect topics and its group — separate from any tenant. **Edge cases / trade-offs:** - **Resolution per worker:** every worker running a connector must reach the secret store; a missing secret on one node fails that connector there. - **Admin client:** if connectors create topics, also consider `admin.override.*`; otherwise topic creation uses the worker identity. - **Rotation:** prefer a managed provider with TTL so rotating a tenant credential triggers reload without manual edits. - **Operational complexity:** per-connector principals multiply credentials, ACLs, and quotas — automate provisioning (e.g. Terraform/operator) or the toil becomes the failure mode. - **Alternative architecture:** if isolation needs are strong, run a Connect cluster (or worker pool) per tenant instead of multiplexing — simpler blast radius at higher resource cost. The override-policy approach is the lighter-weight multiplexed option. **Why this is the right shape:** Security lives in defense-in-depth — override policy limits what configs change, ConfigProvider keeps secrets out of state, broker ACLs enforce least privilege at the data plane, and REST auth protects the control plane. No single layer is sufficient alone.