In Jenkins, what does the controller do compared with an agent, and why is running build steps on the built-in node treated as an anti-pattern?

## Two roles, one system Jenkins is a distributed build system with a deliberately asymmetric topology. Exactly one **controller** runs the Jenkins web application. It renders the UI, exposes the REST API, stores configuration, evaluates pipeline definitions (the Jenkinsfile is interpreted on the controller, step by step), maintains the build queue, and decides which queue item goes to which executor. Around it sit any number of **agents**: separate machines, VMs, containers or pods running the Jenkins agent process, each offering some number of **executors**. An executor is one slot for one build at a time; a `sh` step runs on the agent, in a workspace directory on that agent's disk. The controller and agent talk over a bidirectional channel. Two connection directions exist. **Inbound** agents dial the controller — historically over the JNLP TCP port, and since Jenkins 2.217 usually over WebSocket on the normal HTTP(S) port, which is what makes agents behind NAT and in Kubernetes practical. **Outbound** agents are launched by the controller over SSH. Either way, once connected, the controller sends the agent serialized work and receives results back. ## What lives on the controller `JENKINS_HOME` is the whole state of the installation on one directory: - `jobs/<name>/config.xml` — every job definition, and `builds/` with the history and logs. - `plugins/` — installed plugin code. - `credentials.xml` — the credentials store, encrypted. - `secrets/master.key` and `secrets/hudson.util.Secret` — the key material that decrypts it. - `users/`, `nodes/`, `secrets/initialAdminPassword` and the rest of the configuration surface. The critical point: the encrypted credentials **and** the key that decrypts them sit in the same directory tree. Encryption at rest protects a stolen backup; it does not protect against a process running on the controller with read access to that directory. ## Why building on the built-in node is an anti-pattern Jenkins ships with a built-in node (labelled `built-in`; before Jenkins 2.307 it was called the master node) that can, by default in older installations, execute builds directly on the controller machine. That is convenient and dangerous, for three separate reasons. **Security.** A pipeline step is arbitrary code written by whoever can edit a Jenkinsfile. Running on the controller, `sh 'cat $JENKINS_HOME/secrets/master.key'` is enough to begin decrypting the entire credentials store — including credentials that Jenkins would never have bound into that job. It can also rewrite `config.xml` files to grant itself permissions, or install a plugin. The credential scoping and per-job binding you carefully configured become advisory. **Stability.** The controller must stay responsive for every job, every user and every agent connection. A build that allocates a 12 GB heap, fills the disk with a Docker layer cache, or forks a thousand processes takes down the orchestrator for everyone, not just its own build. Workspaces on the controller also grow inside `JENKINS_HOME`, competing for the same disk as build history. **Scalability.** One controller cannot grow horizontally. Pushing execution onto agents is what lets you add capacity, offer several platforms, and give each build a clean machine. ## The standard hardening ```groovy // Anti-pattern: pins the pipeline to the controller pipeline { agent { label 'built-in' } stages { stage('Build') { steps { sh 'make' } } } } ``` The mitigations, in order of value: 1. **Set the built-in node's number of executors to 0** (Manage Jenkins → Nodes → Built-In Node → Configure). Nothing can be scheduled there any more; queue items wait for a real agent. This is the single most-cited Jenkins hardening step. 2. **Keep Agent → Controller Access Control enabled.** This subsystem restricts which commands and file paths an agent may ask the controller to execute or read over the channel — it is what stops a compromised agent from reaching back into `JENKINS_HOME`. 3. **Label agents and route deliberately**, so jobs express what they need (`linux`, `docker`, `windows`) rather than landing wherever there is capacity. 4. **Prefer short-lived agents** — a container or pod per build — so a build cannot leave state behind for the next one. Persistent agents accumulate caches, credentials in `~/.docker/config.json`, and half-finished checkouts. ## The honest exception A handful of things legitimately run on the controller because they have nowhere else to run: the pipeline's own flow logic (the Groovy CPS interpreter is a controller-side execution), `stash` storage, and lightweight steps in a pipeline-level `post`. That is why `agent none` does not mean "run on the controller" — flow control has always been controller-side, and it is the *workspace* work you are keeping off the box. ## How this comes up in interviews Usually as a scenario: "your Jenkins gets slow and occasionally OOMs during nightly builds" or "a developer exfiltrated production credentials through a Jenkinsfile — how?" Both answers run through the same split. Say what the controller owns, say that build code is untrusted code, and name the zero-executor fix.