How would you diagnose and confirm a classloader leak in a running JVM?

## Step 1 — Confirm the symptom (metaspace, not heap) The defining signature of a classloader leak is **metaspace that grows and never recovers**, especially across redeploys. - **Monitor metaspace**: `jstat -gc <pid>` (look at the `MC`/`MU` metaspace capacity/used columns), or `jstat -gcmetacapacity`. Modern setups read it from JMX/Micrometer or **Java Flight Recorder (JFR)**. - **The fingerprint**: usage climbs in steps after each redeploy and never falls back. Eventually: `java.lang.OutOfMemoryError: Metaspace`. If instead you see 'Java heap space', it's a different (data) leak. - **Size the cap deliberately**: set `-XX:MaxMetaspaceSize` so the leak fails fast and visibly rather than slowly consuming native memory; without a cap, metaspace can grow until the OS kills the process. ## Step 2 — Watch class unloading directly - **`-Xlog:class+unload`** (JDK 9+; older: `-XX:+TraceClassUnloading`) logs each class as it is unloaded. After a redeploy you *expect* a burst of unloads of the old version's classes. **Silence is the smoking gun** — the old classes aren't being unloaded, i.e. the old loader is pinned. ## Step 3 — Take a heap dump - **On OOM automatically**: `-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path`. - **On demand (live objects only)**: `jmap -dump:live,format=b,file=heap.hprof <pid>`. The `live` flag forces a full GC first, so anything remaining is genuinely reachable — perfect for proving a leak. ## Step 4 — Analyze the dump Using **Eclipse MAT** (the standard tool) or **VisualVM**: 1. **Count classloaders**: look for **multiple live instances** of your app/web-app classloader class (e.g. `WebappClassLoader`, `ParallelWebappClassLoader`). After N redeploys you may see N+1 of them — each pinned generation is one leak. 2. **Spot duplicate classes**: the same application class loaded many times (once per stale loader) confirms the buildup. 3. **Path to GC roots**: select a stale classloader and run **'Merge Shortest Paths to GC Roots' → exclude weak/soft/phantom references**. The remaining **strong** path *is* the leak. Typical endings: a `Thread` (→ ThreadLocalMap → entry), a static field of a shared library, a `DriverManager` driver entry, a registered MBean/listener/shutdown hook. 4. MAT's **'Leak Suspects'** report and the **dominator tree** help quantify how much each loader retains. ## Step 5 — Cross-check live - **JFR** can record metaspace and old-object-sample events over time, useful in production where you can't easily pause for a dump. - **async-profiler** / allocation profilers help if the leak co-occurs with allocation churn, though metaspace leaks are primarily about retained class metadata, not allocation rate. ## Step 6 — Fix and verify Sever the single strong reference the path-to-GC-roots revealed (remove the ThreadLocal, stop the thread, deregister the driver/listener, null the static cache). Then **re-run the redeploy loop** and confirm with `-Xlog:class+unload` that the old classes now unload and metaspace returns to baseline. ## Common false leads - A genuinely *growing* class set (e.g. heavy runtime code generation, lots of dynamic proxies/lambdas, scripting) can fill metaspace **without** a classloader leak — distinguish 'too many classes by design' from 'old loaders not unloading' by checking whether usage *recovers* after redeploy. - A too-small `MaxMetaspaceSize` can OOM under legitimate load; verify the ratchet-across-redeploys pattern before blaming a leak.