What is plugin.path in Kafka Connect, and how does the runtime isolate connector plugins? What problems does misconfiguring it cause?
answer
- plugin.path = dirs of plugins
- one classloader per plugin
- point at parent dir, not the JARs
- converters & SMTs are plugins too
- isolation avoids dep clashes
basics
~10 splugin.path is a list of directories where the worker finds connector/converter/transform JARs. Connect loads each plugin in an isolated classloader to avoid dependency conflicts. Misconfiguring it causes 'class not found' errors or dependency clashes.
solid answer
~40 s`plugin.path` is a worker property listing filesystem directories that contain Connect plugins (connectors, converters, SMTs, predicates), each typically packaged as its own subdirectory of JARs (an 'uber-jar' or a directory of dependencies). Connect uses **classloading isolation**: each plugin gets a dedicated `PluginClassLoader` so one connector's dependency (say a particular Jackson or AWS SDK version) can't conflict with another's or with Connect's own. The plugins must sit in subdirectories of a `plugin.path` entry, not on the main `CLASSPATH`. Common misconfigurations: pointing at the directory *of* JARs rather than its parent, leaving plugins on the classpath (defeating isolation and causing version clashes), or forgetting converters/transforms are also plugins. Symptoms are `ClassNotFoundException`/connector class not found at `POST /connectors`, or `NoSuchMethodError`/`LinkageError` from leaked dependencies.
go deeper
Know plugin.path tells the worker where to find connector JARs.
Explain the directory-of-subdirectories layout and that converters/SMTs count as plugins.
Explain per-plugin classloader isolation and the dependency-conflict problems it prevents/causes.
Define the org's plugin packaging/deployment standard (container mounts, isolation, version governance).
## plugin.path and classloader isolation ### What a 'plugin' is In Connect, a **plugin** is any pluggable component the runtime loads: connectors (source/sink), **converters** (JSON/Avro/Protobuf serializers), **transforms** (Single Message Transforms, SMTs), and **predicates**. Each is packaged as a set of JARs. ### What `plugin.path` is `plugin.path` is a comma-separated list of **directories** in the worker config, e.g. `plugin.path=/usr/share/java,/opt/connectors`. Inside each listed directory, every **immediate subdirectory** (or uber-jar) is treated as one plugin and gets its own classloader. So the layout is: ``` /opt/connectors/ <- listed in plugin.path debezium-postgres/ <- one plugin *.jar s3-sink/ <- another plugin *.jar ``` ### Classloader isolation (why it exists) Different connectors bundle conflicting transitive dependencies (different versions of Jackson, Netty, guava, AWS/GCP SDKs). If they all loaded into one flat classpath, you'd get version clashes and runtime `NoSuchMethodError`/`LinkageError`. Connect solves this with a **`PluginClassLoader` per plugin**: each plugin's classes and dependencies are isolated; only a small set of shared API/framework classes come from the parent loader. This lets two connectors use incompatible library versions side by side. ### Misconfiguration symptoms - **Pointing `plugin.path` at the JAR directory itself** instead of its parent: Connect scans subdirectories, so the plugin isn't discovered → 'Failed to find any class that implements Connector' / `ClassNotFoundException` when you `POST /connectors`. - **Putting plugins on the main `CLASSPATH`** (the old pre-isolation approach): they load in the shared loader, breaking isolation and causing dependency clashes between connectors. - **Forgetting converters/SMTs are plugins**: if your Avro converter lives only on the classpath but the connector is isolated (or vice versa), you can hit class-visibility errors. - **Multiple copies of the same plugin** across `plugin.path` entries → ambiguous loading / version confusion. ### Operational notes - At startup the worker logs the plugins it discovered; checking that log is the fastest way to confirm a plugin is on `plugin.path`. - Converters referenced in `key.converter`/`value.converter` must be discoverable as plugins too. - In containers, mount connector JARs into a `plugin.path` directory rather than baking them onto the classpath.
- Why does Connect use a separate classloader per plugin instead of one shared classpath?To isolate each plugin's transitive dependencies, so connectors bundling conflicting library versions (e.g., different Jackson/AWS SDK versions) can coexist without LinkageError/NoSuchMethodError clashes.
- You POST a connector and get 'Failed to find any class that implements Connector'. What's the likely cause?The plugin isn't on plugin.path — often plugin.path points directly at the JAR folder instead of its parent directory, so Connect's subdirectory scan never finds the plugin. Fix the path and restart the worker.
- Saying you should add connector JARs to the main CLASSPATH (that defeats isolation).
- Treating plugin.path as a single JAR file rather than directories of plugin subdirectories.
- Forgetting that converters and SMTs are also plugins loaded from plugin.path.