You have roughly 5,000 servers to keep configured and you also need ad-hoc commands across them. What does adopting Salt's persistent minion agent buy you over a purely SSH-push approach, and what does it cost?

## Frame the question as a cost model, not a feature list The honest comparison is about where the connection cost lives and what capabilities that unlocks. Everything else follows. **Push over SSH.** Every run: open TCP, negotiate, authenticate, ship the payload, execute, collect, close — per host. Parallelism is forks or threads on the control node, so 5,000 hosts means tuning fork counts and living with a long tail. There is no persistent channel, so the control node only knows what a host is doing while it is holding a connection to it. **Persistent agent.** `salt-minion` connects out to the master once and stays connected. A job is a single publish; each minion evaluates the target itself and returns its result. Fan-out is close to constant work for the master, and the wall-clock time is dominated by the slowest minion rather than by connection setup multiplied by fleet size. At 5,000 hosts that difference stops being an optimisation and becomes the difference between a usable and an unusable tool for ad-hoc work. ## What the persistent connection unlocks This is the half candidates usually miss. A live bidirectional channel is not just faster — it makes the minion an active participant: - **Events flow upward.** Beacons on the minion emit events; the master's reactor acts on them. That is closed-loop remediation with detection at the edge, and it has no equivalent in a model where the control node must initiate everything. - **The minion can converge itself.** The minion-side scheduler runs `state.apply` on an interval with a splay, so drift correction does not depend on anyone reaching in — useful for hosts that are frequently offline or behind restrictive networks. - **No inbound access needed.** Minions dial out, so hosts behind NAT or with no inbound SSH still participate. With push, connectivity to every host is a hard prerequisite. - **Real-time inventory.** `grains.items` across the fleet answers "which hosts are on this kernel" in seconds, because the channel is already open. ## What it costs 1. **Agent lifecycle.** A daemon on every host has to be installed at build time, upgraded, and monitored — and a minion that is dead is invisible, so "no return" now has two meanings: the host is down, or the agent is. Version skew between master and minions is a real operational chore. Push tools genuinely have less to go wrong here, which is their main selling point. 2. **Key management.** Keys must be accepted, and that step is where you decide whether an unknown host may join. Auto-accept is convenient and a bad idea in most estates; without it, provisioning has to include a key-acceptance step. 3. **Blast radius.** This is the argument that carries weight at a senior level. The master can execute arbitrary commands as root on every minion, with no per-host credential in the way. Whoever controls the master — or its reactor rules, or the pillar tree — controls production. It deserves the controls you would put on a CI signing key: restricted access, audited change to the state and reactor repositories, and a hard look at who can publish fleet-wide. A push model distributes that authority into SSH credentials, which is more auditable per host and harder to fully centralise. 4. **Master availability and scale.** The master is now on the critical path for both ad-hoc work and, if you use the reactor, remediation. At 5,000 minions you plan for it: multiple masters, or `salt-syndic` to tier masters under a top-level master so publishes fan out through a tree instead of one process. ## Salt's own hedge: `salt-ssh` Salt does not force the choice. `salt-ssh` runs the same state system over SSH with no minion installed, using a **roster** instead of accepted keys: ```yaml web1: host: 10.0.1.10 user: deploy sudo: True ``` You get states, pillar and execution modules without a daemon — and you give up exactly what the bus provided: the constant-time fan-out, events, beacons and the reactor, and the minion scheduler. It is the right tool for appliances you cannot install software on, for bootstrapping the minion itself, and for a small number of hosts where the agent is not worth it. ## The answer that lands Say that at 5,000 hosts with a genuine ad-hoc requirement, the agent's constant-time fan-out and the event bus are worth real money, and that the price is a daemon fleet plus a master that is a root-everywhere trust concentration you must govern deliberately. Then note that a mixed estate is normal: minions where you own the image, `salt-ssh` for the edges. Refusing to name the security cost is what makes this answer sound junior.