Reliability & Failure Domains

An availability commitment is a contract term, not a promise the service stays up. If the provider's own measurement falls short, the remedy is a service credit against your bill for that service, which you normally have to claim.

Instances in the two healthy zones keep serving, but only if the traffic entry point health-checks the dead ones out, the writable data copy is not stranded in the lost zone, and the survivors already had spare capacity.

The control plane - the platform's management API that creates, changes and deletes resources - is degraded, while the data plane that carries request traffic keeps serving. Launching, scaling, replacing and failing over stop; already-running capacity does not.

A replica copies committed writes, so it reproduces the mistaken delete as faithfully as any other write. Only a point-in-time restore rewinds the data to a moment before the mistake, at the cost of a much longer recovery time.

Only the refresh path is broken. A workload still holding an unexpired short-lived credential keeps calling platform services normally. Serving stops at the expiry cliff, when that credential's remaining lifetime runs out — often for many workloads within the same few minutes.

The figure is a ratio of eligible time or eligible requests the provider's own instrumentation judged available, over the eligible total in a fixed window — usually a calendar month, per service and per region. Averaging over that window hides short, severe outages.

Each survivor is asked for about 105% of its own capacity, so the service queues, slows and sheds load at the worst moment. Surviving one zone loss means capping each zone's peak utilisation at (N-1)/N — roughly 66% with three zones.

Availability commitments attach to a deployment configuration, not to a service name. The headline figure normally requires redundant instances spread across separate availability zones, so a single instance in one zone earns a lower commitment or none at all.

Promoting the in-zone synchronous standby loses no acknowledged writes, because every commit already waited for it. Promoting the cross-region asynchronous copy loses whatever the lag held — seconds or minutes of confirmed orders that exist nowhere else.

Accept a frozen fleet and manage demand instead of capacity: keep the durable queue absorbing arrivals, shed or defer the lowest-value work, freeze anything that could shrink or replace workers, and communicate delay rather than failure.

A restore returns rows, not a running service. The name clients resolve still points at the old endpoint, the restored instance has its own credentials, and in-flight messages, caches, derived indexes and registrations elsewhere are untouched by it.

Health and reachability are separate facts. Name resolution turns a dependency's name into an address and the metadata service hands a workload its identity and placement, so both sit in front of everything — a healthy database nobody can find or authenticate to is unreachable, and neither hop is usually drawn.

Trust the customers. Telemetry ingestion is itself a shared platform service, so a flat chart is missing data rather than healthy traffic, and a status page is gated on human confirmation. The signal to believe is the one sharing fewest dependencies with the thing it measures.

Announced maintenance, downtime caused by tenant configuration or code, preview and trial tiers, suspension for policy or non-payment, and causes outside the provider's control are normally carved out — so measured downtime is a subset of what your customers actually felt.

Something in the path had exactly one home in the lost zone — a single writable database, a broker node, a lock or leader, a session cache, an outbound path, or a fleet whose spread had quietly drifted into one zone. The path is only as multi-zone as its least-spread hop.

Automate inside a region, where a synchronous standby loses nothing and the move is cheap; require a declaration across regions, where promoting a lagging copy accepts real data loss and the failure signal is often ambiguous. Automation is only safe with independent observers and fencing.

Size headroom to the demand a workload must absorb while it can create nothing, not to average utilisation, and set it per tier rather than fleet-wide. Cheaper than any percentage: mandate that every failover path completes without creating a resource.

Derive each number from the consequence of losing that service, not from what the platform already does. A ledger holds originals nothing can reconstruct, so it buys a tight recovery point; a reporting copy is derived data and can be rebuilt, so it buys a loose one cheaply.

Independence is capped by the least independent thing underneath, so decide per dependency: accept and document it, move it off the request path, define a deliberate degraded mode, or genuinely replace it. The lead's deliverable is that decision plus an honest availability story, not the removal of a dependency that is not yours.

Less than the weakest link. Serial dependencies multiply, so a request path through three committed services promises the product of their figures — and the only ways to raise it are removing a link from the path, masking one with redundancy, or degrading without it.

Frame it as recovery time and acceptable data loss priced against the cost of each posture. Multi-zone is cheap and lossless; a second region costs duplicated capacity, duplicated data and continuous engineering to keep both true, and rises steeply from cold standby to active-active.

Scale-in on a falling backlog, terminate-and-replace supervision, a deployment already in flight, and scheduled teardown can each destroy capacity you cannot rebuild. Suspend scale-in, switch replacement to drain-not-terminate, and pause deployments until creates succeed.

The restore's real duration grew roughly with the data, so the recovery time objective is now missed although nobody edited it. The recovery point is usually unchanged, because it follows capture frequency rather than volume.

A status page is a published statement, not a live feed: the provider must detect, confirm and scope the impact before anyone posts, and pages are per service and per region rather than per tenant. Use your own out-of-band measurement as the trigger instead.