skip to content

How do you configure a Kafka Connect worker to connect to brokers secured with TLS and SASL, and which prefixes apply to which client?

level: juniorimportance: must knowfreq 70%

answer

  1. security.protocol=SASL_SSL at top level
  2. producer. / consumer. / admin. prefixes
  3. sasl.jaas.config is inline, not a file
  4. top-level = all clients incl. worker internal
  5. truststore trusts broker; keystore = mTLS identity

basics

~20 s

Put the broker security settings (security.protocol, SASL/SSL configs) in the worker properties file. By default they apply to all of Connect's internal clients. Prefix with producer., consumer., or admin. to target a specific client type.

solid answer

~40 s

Kafka Connect runs several internal Kafka clients: a producer (sink offsets / source records), consumer (sink connectors), and admin (creating internal topics). To reach a secured cluster you set top-level keys in the worker config such as security.protocol=SASL_SSL, sasl.mechanism=SCRAM-SHA-512, sasl.jaas.config, and the ssl.truststore.location/password. These top-level keys configure all Connect clients including the worker's own group-membership and internal-topic clients. To override settings for just one client you prefix them: producer.<config>, consumer.<config>, admin.<config>. For source connectors the producer prefix matters; for sink connectors the consumer prefix. The same security keys also need the bootstrap.servers pointed at the secured listener. JAAS is supplied inline via sasl.jaas.config rather than a separate file.

go deeper

for a junior

Know that security settings go in the worker properties, and that there are producer./consumer./admin. prefixes.

for a middle

Explain which client each prefix targets and that top-level keys cover the worker's internal clients.

for a senior

Reason about per-client principals, the internal-topic client gotcha, and SASL_SSL vs mTLS trade-offs.

for a principal

Design a credential/principal strategy across source/sink/admin clients and the worker, with rotation and least privilege.

**What Kafka Connect is:** Connect is a framework for streaming data between Kafka and external systems using reusable plugins called connectors. A *source* connector reads from an external system and writes to Kafka; a *sink* connector reads from Kafka and writes out. Connectors run inside JVM processes called *workers*. **Why there are multiple clients:** A worker doesn't talk to Kafka as a single client. It opens: (1) a **producer** to write source-connector records and to write offsets/status/config to internal topics; (2) a **consumer** for sink connectors to read their input topics; (3) an **admin** client to create internal topics and inspect the cluster; and (4) the distributed worker's own group-coordination client. When the broker is secured, every one of these must present valid credentials. **TLS (SSL) settings** encrypt the connection and let the client verify the broker's certificate. Key configs: `security.protocol` (one of PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL), `ssl.truststore.location` and `ssl.truststore.password` (to trust the broker cert), and for mutual TLS `ssl.keystore.location`/`ssl.keystore.password`/`ssl.key.password`. **SASL settings** authenticate the client to the broker. Key configs: `sasl.mechanism` (e.g. PLAIN, SCRAM-SHA-256/512, GSSAPI for Kerberos, OAUTHBEARER) and `sasl.jaas.config`, an inline JAAS string like `org.apache.kafka.common.security.scram.ScramLoginModule required username="connect" password="...";`. **Where they go and prefixing rules:** Top-level keys in the worker `.properties` (or env vars in container images) apply to *all* Connect clients. To target a specific client, prefix the key: - `producer.security.protocol`, `producer.sasl.jaas.config`, ... → the producer. - `consumer.<key>` → sink consumer. - `admin.<key>` → admin client. This matters when, for instance, source and sink connectors need different identities, or you want a dedicated principal for the admin client that creates internal topics. **Edge cases:** the worker's own internal client (for the offset/config/status topics in distributed mode) follows the *top-level* settings, not the producer/consumer prefixes; if you only set `producer.*` you'll authenticate source records but the worker itself may fail to read its config topic. Always set the top-level security block too.

  • If you set only producer.* security configs, why might the distributed worker still fail to start?
    The worker's own internal client that reads/writes the config, offset, and status topics uses the top-level security settings, not the producer. prefix. Without top-level configs it can't authenticate to those internal topics.
  • How do you supply SASL credentials — JAAS file or inline?
    Kafka clients support inline sasl.jaas.config, which is the idiomatic approach for Connect. You don't need a separate java.security.auth.login.config file, though it still works via the JVM system property.

saying these in an interview costs you the question

  • Saying you must use a separate JAAS file (inline sasl.jaas.config is standard)
  • Claiming producer.* settings cover the worker's internal config/offset/status topic clients
  • Confusing truststore (trust broker cert) with keystore (your own mTLS identity)
  • Thinking PLAINTEXT vs SSL is the same axis as SASL (security.protocol combines both)

context

resolving links…