Walk through a single Puppet agent run, from the agent waking up on its interval to the node actually changing. Where is the catalog compiled, and what goes into it?
answer
- agent asks, server answers
- facts up, catalog down
- Facter plus Hiera, per node
- conditionals resolved before it leaves the server
- report posted at the end
basics
~20 sOn each run interval the Puppet agent collects Facter facts and sends them to the Puppet Server, which compiles a node-specific catalog from the manifests and Hiera data for that node. The agent applies that catalog locally, then posts a report back.
solid answer
~50 sA Puppet agent run is a request/response cycle, not a push. On its run interval — 30 minutes by default — the agent collects facts about itself with Facter (OS, IP, memory, plus any custom facts) and sends them to the Puppet Server with its certificate name. The server classifies the node, evaluates the manifests and modules for that node's environment, and resolves whatever data the classes need through Hiera. What it returns is a **catalog**: a node-specific, fully resolved graph of resources and relationships with no conditionals or variables left in it. The agent walks that graph and, for each resource, asks the relevant provider whether the node already matches; only out-of-sync properties get changed. Finally the agent posts a report of every event. Compilation happens on the server, enforcement happens on the node — that split is the answer interviewers are listening for.
code
bash · 8 lines# Force one foreground run now instead of waiting for the interval
sudo puppet agent -t
# See exactly what the server compiled for this node
sudo puppet catalog download
# What does Facter send up?
facter -p os networking.ipgo deeper
Be able to say Puppet pulls: the agent sends facts up, gets a catalog back, applies it, and repeats on a schedule. Know that Facter supplies the facts.
Explain where compilation happens and why it matters — the server resolves every conditional, variable and Hiera lookup for that one node, so the catalog that arrives is already concrete.
Show you can debug a run from its report: distinguish a compile failure from an apply failure, know that a failed resource skips its dependents, and explain what usecacheonfailure hides.
Own the cost and control model: compilation is server CPU, so interval, splay, environment count and Hiera depth are capacity decisions, and code promotion between environments is your change-control boundary.
## The shape of a run Puppet's classic topology is an agent on every managed node and one or more Puppet Servers (the "master"). Nothing is pushed. The agent daemon wakes on its `runinterval` — 30 minutes out of the box — and initiates a run over HTTPS, authenticating with the client certificate it was issued when it first checked in and an administrator signed it. Everything below happens inside that one request/response cycle, and then the agent goes back to sleep. ## Step 1 — the agent describes itself with facts Before asking for anything, the agent runs **Facter**, which gathers structured facts about the machine: `os`, `networking`, `memory`, `kernel`, `virtual`, and any custom or external facts you have shipped. Those facts are uploaded to the server as part of the request. This is why the same code base can produce different configuration per node — the code is written once, and the facts are what make it specialize. ## Step 2 — the server classifies and compiles The server decides which classes apply to this node. Classification comes from node definitions in `site.pp`, from an external node classifier (ENC), or from a console/node-group in Puppet Enterprise. It then evaluates the Puppet code for that node's **environment** (a directory of modules and a `manifests/` entry point — `production` by default), with the node's facts as top-scope variables. Data lookup happens here too. **Hiera** is Puppet's hierarchical data store: a layered YAML/JSON hierarchy, usually keyed by fact values, that supplies class parameters so code and data stay separate. ```yaml # hiera.yaml hierarchy: - name: "Per-node data" path: "nodes/%{trusted.certname}.yaml" - name: "Per-OS data" path: "os/%{facts.os.family}.yaml" - name: "Common data" path: "common.yaml" ``` The crucial consequence: every `if`, every `case`, every Hiera lookup, every template render is resolved **on the server, at compile time**. The agent never evaluates Puppet language. ## Step 3 — the catalog comes down The output is a catalog: a JSON document listing the concrete resources this one node should have — files with literal content, packages with literal versions, services with their desired state — plus the edges that order them. It is a directed acyclic graph, not a script. The agent caches it locally. ## Step 4 — applying through the RAL The agent walks the graph in dependency order. For each resource it uses Puppet's **resource abstraction layer**: a type (`package`, `file`, `service`, `user`, `exec`) backed by a platform-specific provider (`yum`, `apt`, `systemd`, `launchd`). The provider is asked what the current value of each managed property is, and only properties that differ are written. That read-then-write-only-if-different behaviour is where idempotence comes from — a second run on an already-correct node changes nothing and reports no events. Unmanaged properties are left alone entirely. Puppet does not describe the whole machine; it describes what your catalog declares. ## Step 5 — the report The agent posts a report listing every resource, its status (unchanged, changed, failed, skipped) and the events it generated. Reports are typically forwarded to **PuppetDB**, which also stores facts and catalogs and makes them queryable across the fleet. ## The failure modes worth naming - **Compilation failure** (a syntax error, a missing Hiera key, a bad classification) means no catalog is produced. By default `usecacheonfailure` is true, so the agent falls back to its last cached catalog rather than doing nothing — which is why a broken commit can appear to have "no effect" on nodes for a while. - **A failed resource** does not abort the run. Its dependents are skipped, and unrelated resources still apply — so a run can be half-applied and the report is the only place that shows it. - **Fleet timing**: agents are independent. `splay` and `splaylimit` scatter start times so several hundred nodes do not hit the server in the same second, because catalog compilation is real CPU on the server. ```ini # /etc/puppetlabs/puppet/puppet.conf [agent] runinterval = 30m splay = true splaylimit = 15m ``` To force a run now, an operator uses `puppet agent -t`, which does one foreground, verbose run instead of waiting for the interval.
- What changes if you run masterless with `puppet apply`?`puppet apply` compiles and applies on the same machine: the code and Hiera data must already be on the node (shipped by git, rsync or an image), and Facter runs locally. You lose the certificate-based server, central reporting, and anything that needs PuppetDB, such as exported resources. In exchange there is no server to scale or keep available, which is why immutable-image and small-fleet setups often prefer it.
- A commit breaks compilation for a group of nodes. What do those agents actually do on their next run?They fail to get a new catalog and, because `usecacheonfailure` defaults to true, fall back to the catalog they cached from the last successful run. So they keep enforcing yesterday's configuration and look healthy at a glance — the breakage shows up only in the run reports or as nodes whose catalog timestamp has stopped moving. That is a good argument for alerting on report failures, not just on node liveness.
- Why would you set `splay` on a large fleet?Because catalog compilation is CPU work on the Puppet Server, and agents that were installed or restarted together tend to run in lockstep. `splay` with `splaylimit` scatters each agent's start time by a random offset within a window, flattening the request spike. It costs nothing in correctness — the interval is a floor for how fast a change propagates, not a schedule anyone depends on.
The node sends its measurements up, the server cuts a made-to-measure parts list for that one node, and the node fits the parts itself — the server never touches the machine.
saying these in an interview costs you the question
- Says the Puppet Server pushes changes out to nodes
- Thinks manifests are shipped to the node and evaluated there
- Believes one catalog is compiled and shared by all nodes
- Assumes Hiera lookups happen on the agent at apply time
- Thinks one failed resource aborts the entire run