Control Plane Dependency

## The two planes Every large cloud platform is built as two systems that fail independently, and the whole of this topic follows from that split. The **data plane** is the part that carries your work: the processes already running on rented machines, the disks already attached to them, the packets already flowing along routes that were installed earlier, the bytes already sitting in an object store and being read. Its work is executed close to where the state lives, by components that already hold the configuration they need. The **control plane** is the management API: the interface through which resources are created, changed and deleted. A console, a command line and an infrastructure tool are all just clients of that same API. Its work is coordinated - a create has to pick a physical host, reserve capacity, allocate addresses, update several inventories and make the result visible consistently across a region. | | Control plane | Data plane | |---|---|---| | What it does | Creates, changes and deletes resources | Carries application traffic and serves stored bytes | | A typical request | Launch a worker; attach a volume; change a routing rule | Serve this request; read this object; deliver this packet | | Where its state lives | Coordinated, often region-wide, strongly consistent | Mostly local to a host, instance or path | | What failure looks like | Errors, long hangs, calls that half-complete | Dropped requests, higher latency, unreachable data | ## What actually stops When only the control plane is degraded, the operations that break are the ones that are secretly management calls: - **Scale-out.** Adding a worker is a create call, so a fleet is frozen at the size it already had. - **Rolling deployment.** Every batch retires instances and creates new ones; a deploy started now can strand itself half-done. - **Instance replacement.** A supervision loop that terminates an unhealthy worker and creates a fresh one performs the destructive half successfully and the constructive half not at all. - **Registering new capacity** behind the entry point in front of a fleet, which is a configuration change on a managed component. - **Configuration a new instance reads at boot**, which frequently comes from a management endpoint rather than from the workload's own data path. - **Failover that provisions or promotes something** - standing up a replacement, promoting a standby, repointing a name - because each of those is a change, not a request. What keeps working is everything that was already in place: instances serving, queues delivering, attached disks reading and writing, existing routes carrying packets, caches answering. ## Why this is the usual shape The asymmetry is deliberate on the provider's side, and it is the same property you are supposed to copy. The data plane is built to be **statically stable**: it keeps doing the last thing it was told to do without needing to ask anything. A host does not consult a central service to keep a process running. A route does not need re-approval to carry the next packet. The control plane cannot be built that way, because its entire job is to change global state consistently, which means coordination, which means a much larger shared failure domain. There is one dangerous asymmetry inside the degradation itself: destructive and idempotent calls often keep landing while creates fail or hang. A fleet can therefore shrink during an incident and be unable to grow back. ## What it means for your design 1. **Assume you can create nothing.** Design every incident response so the mitigation is a change to something that already exists, not a request for something new. 2. **Buy the capacity before the incident.** Pre-provisioned headroom and a warm standby are the only kinds of capacity available while creates are failing. 3. **Make the failover path create nothing.** If it has to provision, promote or register, it depends on exactly the thing that is broken. 4. **Stop your own automation from shrinking you** while you cannot grow back. ## Reading the signal Tell the two apart from your own telemetry rather than from a status page, which usually lags the incident it describes. If your service's request success rate and latency are normal while your management calls error or hang, the control plane is degraded. Say that precisely: `the region is down` implies customers are failing and a whole footprint decision is on the table, while `the control plane is degraded` means customers are fine, your capacity is frozen, and the next decision is about what your own automation is allowed to do.