You operate a sidecar-based service mesh across several clusters and dozens of teams, and the control plane supports only a narrow version skew with the proxies. How would you plan and de-risk upgrading the mesh, given that every proxy in the fleet has to change version?

## Why this is a strategy question and not a runbook In a sidecar mesh the proxy binary is injected when a pod is created, so its version is a property of the pod, not of the platform. Nothing the platform team does to the control plane changes a running proxy. That single fact has three consequences a principal engineer is expected to own: - Upgrading the data plane means **recreating every workload in the estate**, and the platform team does not control when workloads are recreated — application owners do. - Control planes support only a **bounded version skew** with proxies, typically a release or two, and generally expect the control plane to be at least as new as the proxies. So the fleet cannot lag indefinitely; there is a deadline attached to every upgrade. - Your response time to a proxy vulnerability equals your fleet turnover time. If the estate takes six weeks to fully recycle, that is your worst-case exposure window, and it is a number the security organisation deserves to be told before the mesh is adopted, not during the incident. ## The migration shape that works **Run two control planes.** Deploy the new version alongside the existing one so both are serving concurrently. Migration then becomes per-namespace: change which control-plane version a namespace is associated with, then let its pods be recreated. Both versions programme their own proxies, and the two populations interoperate for the duration. **Sequence by blast radius.** Platform-owned and internal namespaces first, then a partner team that has agreed to be the canary, then progressively wider. Do not migrate a whole cluster in one action just because the tooling permits it. **Watch the right signals.** A data-plane regression is visible at the hop, not in the application: success rate and p99 latency per service as seen by the caller's proxy, connection failures and locally generated 5xx, certificate issuance and configuration-push health. Establish a baseline before the first namespace moves, and compare migrated versus not-yet-migrated workloads — with two populations live you have a natural control group. **Keep the rollback cheap.** As long as the old control plane is still running, rolling back a namespace is re-pointing it and recreating its pods. Delete the old version too early and rollback becomes a second fleet-wide restart under incident pressure. The cost of leaving it up for a few extra weeks is trivial by comparison. ## Making the treadmill sustainable A one-off migration is a project; the recurring obligation is what actually strains an organisation. Some practices that hold up: - **Ride the normal deploy.** Teams that ship weekly recycle their pods anyway. If injection always uses the currently designated version, most of the fleet upgrades itself for free. Your active effort is only the long tail: stateful workloads, low-change services, things nobody wants to restart. - **Measure freshness, do not assume it.** Report proxy version distribution per namespace and per team, with an explicit target such as "no proxy more than one minor version behind". Freshness is a platform SLO; without the report you discover the laggards during a CVE. - **Publish the contract.** Teams should know the supported skew, the migration window, and what the platform will do about workloads that miss it — including the possibility of the platform recreating them. - **Budget the cost honestly.** Every upgrade consumes coordination time across dozens of teams. Two or three upgrades a year is a standing tax; if the mesh's value does not clearly exceed it, that is a legitimate finding, not defeatism. - **Rehearse.** A non-production cluster with representative workloads catches the incompatibilities — a changed default, a policy that behaves differently, a removed configuration field — while the cost of finding them is low. ## The structural alternative The reason data-plane models that do not put a proxy in the pod are interesting to a platform owner is precisely this problem: if the proxy is a node-level daemon, upgrading it is ordinary node maintenance the platform team already performs, with no application restarts and no negotiation. If your organisation's dominant mesh pain is the upgrade treadmill rather than the need for rich per-workload L7 policy, that trade deserves to be evaluated on its own terms — against the shared blast radius and relative immaturity it brings. Framing the choice that way, rather than as a feature comparison, is what a principal-level answer looks like here.