skip to content

A Kubernetes pod's JVM image has no jcmd; from an ephemeral debug container with a full JDK, jcmd first finds no JVM, then times out once you target the app container. What is wrong, and how do you attach?

level: seniorimportance: should knowfreq 42%

answer

  1. two failures, two boundaries
  2. first the PID namespace
  3. then the filesystem and the user
  4. which /tmp does the error name?
  5. thread dump in the app's log

basics

~20 s

Without a shared PID namespace the debug container cannot see the JVM. Sharing it fixes that, but a different user cannot reach the JVM's /tmp, so the trigger lands in the wrong one. Run as the JVM's UID and GID.

solid answer

~50 s

The first failure is the PID namespace: a debug container that does not share the app container's process namespace sees no JVM to signal. Targeting the app container fixes that, and the JVM becomes visible, often as PID 1. The timeout that follows is the filesystem and the user. The debug container usually runs as root while the app runs as, say, UID 1000, so the kernel normally refuses it `/proc/1/root` and `/proc/1/cwd`; the attach client falls back to its **own** `/tmp`, drops the trigger file there and waits for `/tmp/.java_pid1`, which never appears. Two pieces of evidence point to it: the error names `/tmp/.java_pid1` rather than `/proc/1/root/tmp/.java_pid1`, and a thread dump shows up in the app container's log, because `SIGQUIT` without a trigger is a thread-dump request. Run the debug container as the app's UID and GID, or share one volume at `/tmp` with the same user.

code

console · 4 lines
console
$ jcmd 1 Thread.print
1:
com.sun.tools.attach.AttachNotSupportedException: Unable to open socket file /tmp/.java_pid1: target process 1 doesn't respond within 10500ms or HotSpot VM not loaded
	...

go deeper

for a junior

Remember that a debug container must share the app container's processes to see the JVM, and that jcmd must run as the JVM's user.

for a middle

Explain why the timeout happens after the PID namespace is shared: the client cannot write through /proc/1/root and falls back to its own /tmp.

for a senior

Diagnose from the evidence: which /tmp the error names, the thread dump in the app's log, and the UID and GID on both sides, then change the user, not the timeout.

for a principal

Decide in advance which UID and GID the team's JDK debug image runs as and where dumps go, so the first attach in an incident works.

## The setup The service runs on a minimal Java runtime that leaves out the JDK's diagnostic tools, so there is no `jcmd` in the app container. The standard move is an **ephemeral debug container**: a temporary container added to the running pod from an image that carries a full JDK. It fails twice, for two different reasons, and each failure has its own evidence. ## Failure one: no shared PID namespace A container has its own **PID namespace** unless something joins it to another. A debug container started without targeting the app container sees only its own processes, so: - `jcmd` with no arguments lists no application JVM; - a PID copied from elsewhere either does not exist in this namespace, where the attach client stops with `non existent JVM pid`, or it names an unrelated process. Targeting the app container puts the debug container into the app container's PID namespace. The JVM is now visible, typically as **PID 1** when it is the container's entrypoint, and `NSpid` in `/proc/1/status` reads `1` because both containers share that namespace. ## Failure two: the timeout Now `jcmd 1 Thread.print` waits about ten seconds and reports `Unable to open socket file /tmp/.java_pid1: target process 1 doesn't respond within 10500ms or HotSpot VM not loaded`. To see why, follow the attach client: 1. It wants the target's temporary directory, `/proc/1/root/tmp`, and uses it only if it can write there. 2. The debug container runs as **root**, the JVM as **UID 1000**. Without extra privileges the kernel refuses a different user access to `/proc/1/root` and `/proc/1/cwd`, so the check fails. 3. The client falls back to the debug container's **own** `/tmp` for both the trigger file and the socket. 4. It sends `SIGQUIT`, which does cross the shared PID namespace. 5. The JVM finds no `.attach_pid1` in its own working directory or its own `/tmp`, treats the signal as a thread-dump request, and prints a full thread dump to **its stdout**. 6. The client polls its own `/tmp/.java_pid1`, which nothing will ever create, until the timeout. ## Reading the evidence | Evidence | What it rules in or out | |---|---| | Error names `/tmp/.java_pid1`, not `/proc/1/root/tmp/...` | The client could not write through `/proc/1/root`: a filesystem or user problem | | A thread dump appears in the app container's log at that moment | The signal arrived, so the PID namespace is shared and the JVM handles `SIGQUIT` | | `id` in the debug container differs from `Uid:`/`Gid:` in `/proc/1/status` | The user mismatch is the cause | | `file should be owned by the current user ...` | The socket was found, but the user is wrong | | `file's group should be the current group ...` | The UID matches, the GID does not | The thread dump is worth noticing for another reason: you already have one, free, in the container log, and `kill -3 1` from any process that may signal the JVM would produce another without any JDK tool at all. ## The fix - **Match the user.** Run the debug container with the app's UID and GID. The same user may read `/proc/1/root`, the trigger lands in the JVM's own `/tmp`, the Attach Listener creates `.java_pid1` there, and the socket passes the client's owner and group check. - **Or share `/tmp` and match the user.** If the app container already mounts a pod volume at `/tmp`, mounting the same volume at `/tmp` in the debug container makes the fallback path the right one. The socket's ownership check still demands the same user, so this is an addition, not a substitute. - **Know what not to change.** Raising `sun.tools.attach.attachTimeout` only waits longer for a socket in the wrong directory, and `-XX:+StartAttachListener` creates the socket at startup in the JVM's `/tmp`, which the debug container still cannot see. ## After the attach: where output lands Commands that write files, such as `GC.heap_dump` or `Thread.dump_to_file`, take a path that the **JVM** opens, in the app container's filesystem. `jcmd 1 GC.heap_dump /tmp/heap.hprof` puts the file in the app's `/tmp`, readable from the debug container as `/proc/1/root/tmp/heap.hprof` by the same user. Plan the path before the command, especially when the app's root filesystem is read-only and only a mounted volume is writable.

  • Why is the JVM's PID inside the shared namespace the one jcmd needs, rather than the node's PID?
    The attach client reads the last `NSpid` entry of `/proc/<pid>/status` and names the socket and trigger after it, because that is the PID the JVM sees for itself and uses for `.java_pid<pid>`. Inside a shared namespace both views agree, often on 1. From the node you pass the host PID and the client resolves the inner one for you.
  • The UID now matches, but jcmd reports that the file's group should be the current group. What next?
    The client found the socket and checked its owner, then its group: the socket's GID must equal the client's effective GID unless the client is root. Set the debug container's group to the app's GID as well. Nothing on the JVM side needs to change.
  • Where should you write a heap dump requested through jcmd from a debug container?
    Somewhere the JVM itself can write, because `GC.heap_dump` opens the path in the app container's filesystem, and somewhere that survives and can be copied out: a mounted volume rather than the container's own layer. Check free space first, since a dump runs to roughly the size of the live heap, and remember the file is created readable only by the JVM's user.

  • Raising the attach timeout will eventually let jcmd connect.
  • Adding -XX:+StartAttachListener fixes attach from any debug container.
  • Running the debug container as root bypasses every attach check.
  • A shared PID namespace is all jcmd needs from a debug container.
  • GC.heap_dump writes the file into the debug container where jcmd runs.