You must apply a config change across 60 web servers with Ansible without taking the whole fleet out at once. Which play-level keywords control that, and how do handlers behave under them?

## The default is all at once By default Ansible walks tasks in order and runs each task across **all** hosts in the play (up to `forks`) before moving to the next. For a config change that ends in a service restart, that means every host restarts at roughly the same moment — a fleet-wide outage delivered efficiently. ## serial: batching the play `serial` is a play keyword that splits the host list into batches. The play runs completely for batch one, then completely for batch two, and so on. ```yaml - hosts: web serial: 5 max_fail_percentage: 0 tasks: - name: Push the config ansible.builtin.template: src: app.conf.j2 dest: /etc/app/app.conf notify: restart app handlers: - name: restart app ansible.builtin.service: name: app state: restarted ``` Accepted forms: - `serial: 5` — fixed batch size. - `serial: "25%"` — proportional to the number of hosts. - `serial: [1, 5, "25%"]` — a ramp: one host first as a canary, then five, then quarters. The last value repeats for the remaining batches. ## Why handlers are the crux Because the **entire play** is re-executed per batch, the end-of-play handler flush happens at the end of **each batch**. This is what turns "change config, notify restart" into a genuine rolling restart: batch one is reconfigured and restarted, then batch two, and so on. Without `serial`, the same playbook queues the handler on every host and restarts all sixty at the end of the single play. Two consequences follow: - Anything that must happen once for the whole run — a database migration, a notification — must be kept out of the batched play, or guarded, because a batched play's tasks execute again for every batch. - Facts gathered and variables set with `set_fact` inside the play are re-derived per batch, since the play genuinely runs again. ## Failing safely: max_fail_percentage By default Ansible keeps going as long as at least one host in the batch survives; a batch is only fatal when every host in it has failed. That is far too permissive for a rolling change. `max_fail_percentage: 0` aborts the whole play if any host in a batch fails, which is normally what you want: stop after the canary breaks, not after 60 machines are broken. A non-zero value tolerates a stated fraction per batch. ## Getting hosts in and out of the pool The usual rolling-update shape brackets the change with load-balancer work, and that work must run somewhere other than the host being drained: ```yaml - name: Drain this host at the load balancer community.general.haproxy: state: disabled host: "{{ inventory_hostname }}" backend: app delegate_to: lb01.example.com ``` `delegate_to` runs the task on a different host while the loop variable and the current host's variables still refer to the host being processed — which is exactly why `inventory_hostname` above names the machine being drained, not the load balancer. `delegate_to: localhost` (or the `local_action` shorthand) runs on the controller, the standard way to call an API. `run_once: true` executes a task a single time per batch on the first host, rather than once per host. ## Ordering and strategy `order` on the play controls how hosts are picked into batches — `inventory` (default), `sorted`, `reverse_sorted`, `shuffle` — which matters when your inventory happens to group all of one availability zone together and every batch would otherwise land in the same failure domain. `strategy` is a different axis. The default `linear` keeps hosts in lockstep: every host finishes task N before any starts task N+1. `strategy: free` lets each host race ahead through the whole play independently, which is faster but destroys the lockstep guarantee — and with it any assumption that hosts restart together. For a rolling change you want `linear` inside batches and `serial` between them, not `free`. `throttle: 2` caps how many hosts run a *single task* at once inside the current batch, useful when one particular step hammers a shared resource without needing the whole play batched more tightly. ## The tradeoff to say out loud Small batches mean a long run: sixty hosts at `serial: 1` is sixty full plays. Large batches finish fast and risk more capacity at once. The honest answer sizes the batch against how much capacity the service can lose and still serve traffic — plus a canary batch of one so a broken config takes out a single host, and `max_fail_percentage: 0` so the run stops there.