Using the Jenkins Kubernetes plugin, how is a build executed, and in a pod template that defines a maven container alongside the default agent container, what decides which container an `sh` step runs in?

## The execution model The Kubernetes plugin makes Jenkins agents disposable. Instead of a fleet of long-lived machines, the controller holds a **cloud** configuration pointing at a cluster, and each build that requests a matching agent causes a **pod** to be created, used once, and deleted. The sequence: 1. A build requests an agent — either by label matching a pod template defined in the cloud configuration, or by declaring the pod inline in the Jenkinsfile. 2. The plugin creates the pod in the configured namespace, with a generated name. 3. One container is the **inbound agent**, named `jnlp` by default. It runs the Jenkins agent JAR, dials the controller (WebSocket over HTTPS in modern setups, or the older TCP JNLP port), and registers as a node. 4. The controller schedules the build onto that node. Steps execute inside the pod. 5. When the build finishes, the node is disconnected and the pod deleted. A declarative pipeline expresses it like this: ```groovy pipeline { agent { kubernetes { defaultContainer 'maven' yaml ''' apiVersion: v1 kind: Pod spec: containers: - name: maven image: maven:3.9-eclipse-temurin-17 command: ['sleep'] args: ['infinity'] resources: requests: {cpu: '500m', memory: '1Gi'} ''' } } stages { stage('Build') { steps { sh 'mvn -B verify' } } } } ``` ## Which container runs the step This is the part candidates get wrong. A pod has several containers, but `sh` has to pick one. The rule: - **By default, steps run in the agent container** (`jnlp`). That image contains a JRE and the agent JAR — no Maven, no Node, no kubectl. A `sh 'mvn -v'` there fails with "command not found", which is the single most common first-day confusion with this plugin. - **`container('maven') { … }`** switches the enclosed steps into that named container. The step is executed there via the Kubernetes exec API. - **`defaultContainer 'maven'`** in the declarative `kubernetes` block makes that container the target for the whole block, so you stop writing `container(...)` around everything. Two constraints follow from how containers are used here. First, every tool container must **stay running** — a container whose image entrypoint exits immediately (most language images do) leaves nothing to exec into. Give it `command: ['sleep']` with `args: ['infinity']`, or equivalent, or enable a TTY. Second, all containers see the same **workspace volume**: the plugin mounts an `emptyDir` (the `workspace-volume`) at the agent's working directory, typically `/home/jenkins/agent`, into every container. That is what makes it legal for a `maven` container to compile and a later `container('kubectl')` block to read the resulting files. ## Failure modes worth knowing - **Pod never schedules.** Requests exceed what any node can offer, or a node selector, taint or quota blocks it. Symptom: the build sits waiting for an executor, sometimes for the plugin's provisioning timeout, then errors. Diagnose with the pod's events, not the Jenkins log alone. - **Container OOMKilled or pod evicted mid-build.** The build dies abruptly, typically with a channel-closed or agent-disconnected error, and because the workspace was an `emptyDir` inside the pod, it is gone. Nothing is recoverable except what you stashed or archived along the way. Right-size memory requests and limits on the tool containers, not just the agent container. - **Image pull failures** show up as an unschedulable-looking hang; a private registry needs an `imagePullSecret` in the pod spec. - **Every build pays a cold start.** Pod creation plus image pull plus agent connection is tens of seconds. `idleMinutes` on a pod template keeps a pod alive for reuse by subsequent builds, and `podRetention` controls whether pods are kept after failure for debugging — both trade away the clean-per-build property in exchange for latency or forensics. ## Why teams choose it The pull is that agent capacity becomes elastic and every build starts from a known image. The dirty-workspace class of bug disappears, because there is no workspace to inherit. Toolchains are declared in the Jenkinsfile rather than installed by hand on VMs, so "works on agent 3, fails on agent 7" stops happening. The costs are equally real: a cold start per build, a dependency on cluster capacity and quota, YAML that is now part of every pipeline's blast radius, and the fact that caches must be deliberately engineered — a persistent volume or a remote cache — because nothing survives the pod. ## Interview framing The question is a good discriminator because the naive answer ("it runs builds in Kubernetes") says nothing. What demonstrates real use is the container-selection rule, the sleeping-tool-container requirement, the shared workspace volume, and knowing that an evicted pod takes the workspace with it.