How do you keep secrets like passwords out of a connector config submitted to the REST API? Explain ConfigProvider externalization.

${provider:path:key} placeholder
config.providers on the worker
FileConfigProvider / Vault / env
placeholder stored, secret resolved at runtime
GET /config shows placeholder, not secret

basics

~20 s

Use a ConfigProvider: put a placeholder like ${file:/path:key} or ${vault:...} in the connector config instead of the literal secret. Connect resolves it at runtime from the provider, so the secret never lives in the REST payload or the config topic, and GET /config shows the placeholder.

solid answer

~40 s

Kafka Connect supports config externalization via the ConfigProvider SPI. You configure providers on the worker (config.providers and config.providers.<alias>.class), then reference values in the connector config with the indirect syntax ${provider:path:key} — for example ${file:/opt/secrets/db.properties:password} using the built-in FileConfigProvider, or a Vault/AWS/env provider. When the connector starts, the worker resolves each placeholder by calling the provider; the resolved secret is held only in memory in the task. Crucially, the unresolved placeholder — not the secret — is what's stored in the Connect config topic and what GET /connectors/{name}/config returns, so secrets never appear in the REST API payload, logs, or internal topics. Some providers support TTL-based refresh so rotated secrets are re-read without recreating the connector. This is the standard way to satisfy secret-management requirements while still managing connectors declaratively over REST.

for a junior

Know that secrets should be placeholders, not literals, in a connector config.

for a middle

Know the ${provider:path:key} syntax and that providers like FileConfigProvider resolve them at runtime.

for a senior

Explain that placeholders (not secrets) are stored/returned, worker-level provider config, and TTL refresh for rotation.

for a principal

Design end-to-end secret management: provider choice per environment, cluster-wide consistency, rotation, plus TLS on the REST listener and access control around GET /config.

The problem. A connector config is a plain string map submitted over the REST API and then persisted to Connect's internal config topic (connect-configs). If you put a database password directly in the config, that secret is now in the REST request, in the durable config topic, and visible via GET /connectors/{name}/config to anyone with API access. That's unacceptable for production secrets.

The solution: ConfigProvider externalization. Connect defines a ConfigProvider SPI (KIP-297). A provider knows how to fetch values from an external secret store at runtime. You wire providers into the worker config:

config.providers=file,vault
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
config.providers.vault.class=com.example.VaultConfigProvider
config.providers.vault.param.address=https://vault:8200

Then in the connector config you use indirect references instead of literals, with the syntax ${provider:path:key}:

"connection.password": "${file:/opt/connect/secrets/db.properties:db_password}"
"api.token": "${vault:secret/data/myconn:token}"

provider = the alias you registered (file, vault).
path = where to look (a file path, a Vault secret path, etc.).
key = which value to pull from that path.

Resolution timing and storage — the key guarantee. The placeholder string is what is submitted, validated, and stored verbatim in the config topic. The actual secret is resolved only at runtime, in memory, when the worker instantiates the connector/task. Therefore:

GET /connectors/{name}/config returns the placeholder, not the secret.
The secret never enters the REST payload, the config topic, or (with proper logging hygiene) the logs.
Rotating the secret means updating the external store, not the connector config.

Built-in and common providers:

FileConfigProvider — reads keys from a properties file on the worker (the canonical built-in example).
DirectoryConfigProvider — reads each file in a directory as a value (handy for Kubernetes secret mounts).
EnvVarConfigProvider — reads environment variables.
Third-party: HashiCorp Vault, AWS Secrets Manager, etc.

TTL / refresh. A provider may return a TTL with a value (via ConfigData). Connect can schedule a reload so that when a secret rotates, the connector picks up the new value without being recreated — important for short-lived credentials.

Edge cases / gotchas:

Providers are configured per worker; every worker in the cluster must have the same providers and access to the backing store, or resolution fails on whichever worker runs the task.
Validation/connectivity checks at config-submit time may need the secret resolvable too, so the provider must be reachable from the worker handling the request.
The indirect syntax must match exactly; a typo yields the literal ${...} string being passed to the connector, which usually surfaces as an auth failure at task time.
This protects against secrets-at-rest and secrets-in-API exposure; it does not by itself encrypt the REST channel — use TLS on the Connect REST listener for transport security.

**The problem.** A connector config is a plain string map submitted over the REST API and then persisted to Connect's internal **config topic** (`connect-configs`). If you put a database password directly in the config, that secret is now in the REST request, in the durable config topic, and visible via **GET /connectors/{name}/config** to anyone with API access. That's unacceptable for production secrets. **The solution: ConfigProvider externalization.** Connect defines a **ConfigProvider** SPI (KIP-297). A provider knows how to fetch values from an external secret store at runtime. You wire providers into the **worker** config: ``` config.providers=file,vault config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider config.providers.vault.class=com.example.VaultConfigProvider config.providers.vault.param.address=https://vault:8200 ``` Then in the **connector** config you use **indirect references** instead of literals, with the syntax `${provider:path:key}`: ``` "connection.password": "${file:/opt/connect/secrets/db.properties:db_password}" "api.token": "${vault:secret/data/myconn:token}" ``` - **provider** = the alias you registered (`file`, `vault`). - **path** = where to look (a file path, a Vault secret path, etc.). - **key** = which value to pull from that path. **Resolution timing and storage — the key guarantee.** The placeholder string is what is submitted, validated, and **stored verbatim in the config topic**. The actual secret is **resolved only at runtime**, in memory, when the worker instantiates the connector/task. Therefore: - **GET /connectors/{name}/config returns the placeholder**, not the secret. - The secret never enters the REST payload, the config topic, or (with proper logging hygiene) the logs. - Rotating the secret means updating the external store, not the connector config. **Built-in and common providers:** - **FileConfigProvider** — reads keys from a properties file on the worker (the canonical built-in example). - **DirectoryConfigProvider** — reads each file in a directory as a value (handy for Kubernetes secret mounts). - **EnvVarConfigProvider** — reads environment variables. - Third-party: HashiCorp Vault, AWS Secrets Manager, etc. **TTL / refresh.** A provider may return a TTL with a value (via `ConfigData`). Connect can schedule a **reload** so that when a secret rotates, the connector picks up the new value without being recreated — important for short-lived credentials. **Edge cases / gotchas:** - Providers are configured **per worker**; every worker in the cluster must have the same providers and access to the backing store, or resolution fails on whichever worker runs the task. - Validation/connectivity checks at config-submit time may need the secret resolvable too, so the provider must be reachable from the worker handling the request. - The indirect syntax must match exactly; a typo yields the literal `${...}` string being passed to the connector, which usually surfaces as an auth failure at task time. - This protects against secrets-at-rest and secrets-in-API exposure; it does not by itself encrypt the REST channel — use TLS on the Connect REST listener for transport security.

If you use ${file:...:password}, what does GET /connectors/{name}/config return for that field?
The literal placeholder string ${file:...:password}, not the resolved secret — the secret is only resolved in memory at runtime and is never stored in the config topic or returned by the API.
Where are config providers configured, and why does that matter in a multi-worker cluster?
On each worker (config.providers / .class). Every worker must have the same providers and access to the backing store, because any worker may run the task and must be able to resolve the placeholders.

saying these in an interview costs you the question

Saying the secret is encrypted in the config topic — it isn't stored there at all; only the placeholder is.
Putting providers in the connector config — they're configured on the worker.
Claiming GET /config returns the resolved secret — it returns the placeholder.
Assuming externalization secures the REST channel — you still need TLS for transport.

How do you keep secrets like passwords out of a connector config submitted to the REST API? Explain ConfigProvider externalization.

answer

basics

solid answer

go deeper

context

How do you keep secrets like passwords out of a connector config submitted to the REST API? Explain ConfigProvider externalization.

§ answer

basics

solid answer

§ go deeper

▾level notes4 levels

▾deep dive~2 min

▾common follow-ups2 questions

▾red flags4 flags

§ context

▾related questions4 questions

▾related topics2 topics

answer

go deeper

context