How does ConfigProvider (e.g. FileConfigProvider) let you keep secrets out of connector configs, and how does the ${...} indirection work?

**The problem:** Connector configs routinely contain secrets — database passwords, broker SASL credentials, API tokens. In distributed mode those configs are stored in the internal **config topic** and are retrievable via the REST API (`GET /connectors/{name}/config`). Storing plaintext secrets there means anyone with topic or REST access reads them. **The abstraction:** `org.apache.kafka.common.config.provider.ConfigProvider` is a pluggable interface. A provider knows how to fetch secret values from some backing store given a *path* and a *key*. Connect ships `FileConfigProvider`, `DirectoryConfigProvider`, and `EnvVarConfigProvider`; vendors/community add Vault, AWS Secrets Manager, GCP Secret Manager, Kubernetes, etc. **Registration (worker config):** ``` config.providers=file,vault config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider config.providers.vault.class=...VaultConfigProvider config.providers.vault.param.address=https://vault:8200 ``` `config.providers` lists logical names; each `config.providers.<name>.class` names the implementation; `config.providers.<name>.param.<x>` passes provider-specific params. **The indirection syntax:** In any connector (or even worker) config value you write `${<name>:<path>:<key>}`. For FileConfigProvider: `${file:/opt/connect/secrets/db.properties:password}` means *use the provider named `file`, open file `/opt/connect/secrets/db.properties`, return the value of key `password`*. For EnvVarConfigProvider the path is typically empty: `${env::DB_PASSWORD}` (or `${env:DB_PASSWORD}` depending on version). Some providers ignore the path; the three-part shape is `${name:path:key}` and a missing path leaves an empty segment. **Resolution timing and where the secret lives:** Substitution happens **on the worker when the connector is instantiated/started**, not when the config is submitted. The stored config (config topic) and REST responses keep the literal `${...}` placeholder — that's the whole point: the secret is never persisted in Connect's state. The provider reads the actual secret from its store at runtime on each worker that runs the connector, so the secret file/secret-manager entry must be present on every worker. **Rotation / TTL:** Some providers return a TTL with each value. Connect can schedule a config-reload (and connector restart) when the TTL expires via the `ConfigChangeCallback`/`subscribe` mechanism, enabling secret rotation without manual reconfiguration. FileConfigProvider itself doesn't watch the file unless wired for it; managed providers commonly do. **Edge cases / gotchas:** - The secret file must exist on *every* worker node and be readable by the Connect process — a missing file on one worker causes that worker to fail starting the connector. - File permissions matter: protect the properties file (chmod 600) — Connect just reads it. - `${...}` is resolved for connector configs and (depending on version) worker configs too; not all positions support it across all versions. - If a provider name in `${name:...}` isn't registered, the value is left as the literal string, often causing a confusing downstream auth failure rather than a clear error. **Security value:** This decouples secret storage from Connect. With a managed provider you get centralized rotation, audit, and access control; even with plain FileConfigProvider you keep secrets off the config topic and out of REST output and source control.