In Salt, how do beacons and the reactor turn something happening on a minion into an automated response, and what goes wrong when the reaction is heavy or self-triggering?

## The bus carries events, not just jobs Everything in Salt already travels over an event bus: job publishes, job returns, minion authentication, key acceptance. Each event has a **tag** (a slash-delimited string) and a data payload. You can watch it live on the master: ```bash salt-run state.event pretty=True ``` Once you accept that the bus exists, event-driven automation is just two additions: something that puts interesting events on it, and something that acts on them. ## Beacons: the minion-side sensor A beacon runs inside `salt-minion` and polls or watches a local condition, emitting an event when it trips. They are configured on the minion — in `/etc/salt/minion` or a drop-in under `/etc/salt/minion.d/`: ```yaml beacons: service: - services: nginx: onchangeonly: True - interval: 10 ``` Commonly used beacons include `inotify` (file or directory changes), `service` (a service's running state), `diskusage`, `load`, `memusage` and `network_info`. `interval` controls how often the beacon is checked. Because beacons live on the minion, detection is local and immediate — nothing on the master is polling thousands of hosts asking "is nginx up?". ## The reactor: the master-side rule engine The reactor is configured on the **master**, mapping tag globs to reactor SLS files: ```yaml reactor: - 'salt/minion/*/start': - /srv/reactor/new_minion.sls - 'salt/beacon/*/service/nginx': - /srv/reactor/restart_nginx.sls ``` A reactor SLS is rendered with the event's `tag` and `data` available, and declares what to do: ```yaml apply_highstate: local.state.apply: - tgt: {{ data['id'] }} ``` The verbs are `local:` (run an execution module on a minion, i.e. what the `salt` CLI does), `runner:` (run a runner on the master), and `wheel:` (master management such as key operations). The `salt/minion/*/start` example is the canonical one: a freshly provisioned host boots, its minion authenticates, the event fires, and the reactor immediately converges it — no operator involvement and no scheduled sweep. ## Where this design pays off The combination gives closed-loop remediation with the detection at the edge and the policy centralised. Because the transport is already there, latency is low, and because the reactor sits on the master, the rules are reviewed in one repository rather than scattered across hosts. This is genuinely the thing Salt does that its contemporaries do not do natively. ## Failure mode one: a heavy reaction blocks the reaction system The reaction system is a component of the master and is explicitly documented as not being the place for long-running work. Put a five-minute orchestration inline in a reactor SLS and you delay every other reaction behind it — which matters most exactly when you need it, during an incident where many minions are firing events at once. The correct shape is to make the reaction *dispatch* rather than *do*: call an orchestrate runner asynchronously and let it own the long job, keeping the reactor's own step short. ## Failure mode two: the loop A beacon watches `/etc/myapp/app.conf` with `inotify`; the reaction applies a state that rewrites `/etc/myapp/app.conf`; the write fires the beacon again. Now you have an infinite loop that will happily saturate a master. Guard against it deliberately: - Have the reaction converge something *other* than what the beacon watches, or make the state genuinely idempotent so the second run writes nothing and produces no inotify event. - Use the beacon's `disable_during_state_run` option where supported, so a beacon does not fire because of changes made by Salt's own state run. - Use `onchangeonly` and sensible `interval` values so a flapping service does not emit an event every second. ## Failure mode three: the stampede A rack loses power. Two hundred minions come back and all fire `salt/minion/*/start`, and your reactor applies a full highstate to each. The master's file server and your package mirror take the hit simultaneously. Reactions that fan out need the same batching thinking you would apply to any fleet-wide job, and it is worth asking whether the reaction should enqueue work rather than execute it immediately. ## What to say in an interview Name the split — beacon on the minion, reactor on the master, event tag as the contract between them — give one real pairing such as auto-highstating a new minion, and then volunteer the operational caveats. Knowing that the reactor should stay lightweight and that self-triggering loops are a real hazard is what separates someone who has run this from someone who has read the feature list.