Across an Android and Apple Appium fleet, how do you upgrade without breaking agent startup?

## What is actually coupled Since Appium 2 the drivers are separately installed extensions rather than part of the server, and each one declares which server line it expects. That is the layer most teams do remember. The layer they forget is the one below it: the **on-device agent is an artefact of the driver that put it there**. Upgrading a driver therefore changes what has to be on every device in the fleet, and the devices do not know that yet. That gives three layers with different upgrade mechanics and, importantly, different blast radii: - the **server**, upgraded on hosts; - the **drivers**, upgraded on hosts but defining what devices must carry; - the **agents**, which live on devices and change only when something puts a new one there. ## The two chains are not the same length | | Android | Apple platforms | |---|---|---| | what the driver depends on | the server line it expects | the server line it expects | | what reaches the device | helper packages installed by the driver | WebDriverAgent built by `xcodebuild` | | extra links you do not own | none | the host's Xcode and the device OS | | when the cost lands | first session on each device | first build on each host, or provisioning | | what a stale device holds | an old agent package | an old installed agent, if you preinstall | The asymmetry is the whole point. On Android the chain ends at the driver: change the driver, and each device picks up what it needs at its next session. On Apple platforms the chain runs on through the host's toolchain and the device's OS, neither of which upgrades on your timetable. A device OS update pushed overnight, or an Xcode bump applied by whoever maintains the build hosts, is an Appium change whether anyone filed it as one. ## Sequencing an upgrade 1. **Pin everything in one place.** The server, each driver, and the step that provisions agents onto devices belong in a single manifest, so "which versions is this fleet on" has one answer rather than one per host. 2. **Move one axis at a time.** Server, then drivers, then the agent-provisioning step — or the reverse — but never two together, because a startup failure with two changes in flight is a bisect you did not need to run. 3. **Stage on a small device set.** A handful of devices covering the OS versions and form factors that differ most will surface a broken agent long before the fleet does, and at a cost you can absorb. 4. **Treat the Apple toolchain as part of the change.** Schedule Xcode and device OS moves deliberately, and re-validate agent startup after each, rather than discovering it through a morning of red runs. 5. **Refresh the devices explicitly.** Make removing or replacing the old agent part of the rollout instead of hoping every session discovers the mismatch and repairs it. 6. **Watch startup, not just pass rate.** Session-creation duration and failure counts, grouped by device and by host, are the signal that says an upgrade landed badly — long before anyone can tell from which tests are red. ## Devices hold state your rollback does not undo The most expensive mistake in this area is treating rollback as a host-side operation. Downgrading packages on the hosts leaves every device holding whatever the newer driver put there, so: - a rollback plan has to name what happens to the agents on devices, not only the packages on hosts; - a device that missed a rollout and a device that missed a rollback look identical from the outside — both fail startup for a mismatch — so the fleet needs to be able to report what each device is actually carrying; - on Apple platforms, if you preinstall agents, the rollback includes re-provisioning them, which takes real time you should have planned for; - "it works on the ones we tested" is exactly the shape of a partially-applied change, and it should be read as a fleet-state question rather than a flaky-device one. ## Where drift comes back Alignment is not a state you reach; it is one you keep losing, and it is worth naming the usual leaks. New devices join carrying nothing, or carrying whatever image they were built from. A host is rebuilt and comes back on a different toolchain. A device OS auto-updates. Someone debugs a problem by upgrading one driver on one host and never reverts it. Each of those is invisible until a session tries to start, which is precisely why agent startup failures are the fleet's early-warning system rather than a nuisance. The judgment a lead is expected to own is that this is a **configuration management problem wearing a testing costume**. The suite is not flaky; the fleet is inconsistent, and the startup step is simply the first place that inconsistency is allowed to show. Investing in knowing what every device carries pays for itself the first time an upgrade goes sideways, because it turns a fleet-wide outage into a list of named devices to reconcile.