Updates & Availability

Stopping all six copies leaves the shared name with no copy able to serve until a replacement finishes starting, so every request in that window fails. Replacing a batch at a time keeps the rest serving throughout.

Because those copies are not interchangeable. Each owns a slice of durable data and a name the replacement must reclaim, and the group needs a majority alive, so only one may be missing at a time.

The signal does not track the pain. A rendering worker that spends most of its time waiting on downloads and uploads shows modest processor use even while items pile up, so the measurement never crosses its target and the copy count never rises.

A revision is a snapshot of the entire declared spec at one apply - image reference, copy count, reservations, ceilings and settings together. Reapplying an earlier revision runs a normal forward rollout whose target happens to be that older spec.

A revision records the desired spec, so reapplying one changes which version runs; it says nothing about rows already committed to a store outside the workload. The spec reverts, the written data does not.

A started process is not yet a serving one, and the step is trading away a copy that can serve. Gating on the readiness answer keeps the exchange honest; gating on process start removes serving capacity in exchange for capacity that does not exist yet.

A drain is a disruption the platform is told about first: it stops placing new work on the host, asks the copies already there to stop one at a time, and waits for replacements. A host failure announces nothing and the copies are simply gone.

Both directions must work at once: the new member reads what the old ones wrote, and the old ones tolerate what the new one writes. They are each other's replication source for the whole rollout.

The drain is blocked, not broken: every removal is a request checked against the cap, and while the workload is already at the floor of available copies, granting one more would breach it, so the request is refused and quietly retried.

It must wait until the replaced copy has rejoined its group and caught up. A started process answers its port long before its store is replayed, so advancing on that gate stacks up members that are behind.

The loop is reacting to its own actions. Adding copies drives the per-copy signal below target, the loop removes them, the signal jumps back above target, and it adds again. A settling window, a tolerance band around the target, and slower shrinking than growth damp the oscillation.

It waits for the whole cold path: something in the request path must notice it, raise the count from zero, and hold the request while a host is chosen, the image is pulled if absent, the process starts and the readiness check passes. That is the cold start.

The arithmetic can never be satisfied: taking the only copy down leaves zero available, below any floor of one, so the removal is refused and retried forever. Resolving it means more copies, a looser cap, or a deliberate outage.

Reactive scaling has a chain of delays: an averaged measurement window, the evaluation interval, placement and image pull, process start-up, then the readiness check and warm-up before traffic is routed. Two to five minutes is normal, so a two-minute spike finishes first.

A revision is the whole spec applied atomically, so reapplying an older one restores every field it holds, including settings that changed after it. Rollback granularity is the apply, never the single field you regret.

Every step's gate opened immediately, so old copies were removed at full speed and the whole set ended up on copies that could not yet serve. The batch structure survived; the property it protects did not, because the ready answer was true too early.

The new copies are not reporting ready, so the guard that allows an old copy to be removed keeps evaluating false and the step never advances. The rollout is stuck on purpose, holding the old copies in service rather than trading them for copies that cannot serve.

Judge it on three things: whether forty minutes is inherent or a defect, whether redundancy still covers an unplanned loss while one member is away, and whether a rollout that long can finish inside any window you can commit to.

Make a cap a contract the workload owner has to fund: reject caps whose floor equals the declared copy count, prefer caps expressed against that count, default the workloads that declare none, and alert the owner - not the operator - when a drain blocks.

Revision history is append-only: going back adds a new highest entry holding the older spec, and retention drops the oldest. Nothing is popped or deleted, and the bad entry stays readable until it ages out.

A cleanly split group: every copy above the hold position runs the new version with its own store intact, every copy at or below it is untouched on the old one. Nothing is half-replaced, and nothing is reverted.

Machines. Replica autoscaling changes a number, not capacity: each copy needs a host with enough unreserved room for its reservation, and the pool ran out. Growing the pool is a second, much slower loop, so host supply is the real ceiling on copy count.

The cap counts copies that report themselves available, and a copy can report available before it has rejoined the store's own membership. Two of three were counted, only one was really voting, so removing the third left no majority.