Orchestration Model

The diff proves intent and nothing else: exactly these two fields are being changed, and the document is the complete statement of what is wanted. Whether six copies fit, what defaults apply unseen, and what is running today are all read elsewhere.

Dedicated hosts are worth it when the workloads genuinely differ in the hardware they need, the interference they can tolerate, or the separation required of them — not when one team simply wants to be treated as more important.

Decide on three axes: the fixed floor each extra cluster carries, the blast radius teams are willing to share, and who absorbs each cluster's operational load. Separation inside one cluster is soft, so a security argument is a different question with a different answer.

It counts clean finishes, not started attempts. The at-once number is only a width limit: the platform keeps roughly that many attempts in flight, tops them up as each one ends, and marks the job finished when 500 have succeeded.

A custom controller must register a declared type, be triggered by an object's identity rather than an event payload, re-read declared and live state on every pass, act idempotently, and report progress instead of failing terminally.

The provider runs the deciding half — the API, its state store and that store's backups — and is the only party who can repair it. Hosts, node agents, workloads and the declared state you put in stay yours.

Completion is a property of the group, not of one member. A run-to-completion group is done only once its members have stopped, so a helper with no exit condition keeps the group alive and the run never records success.

Mass replacement can turn an observation problem into a real outage: the surviving hosts cannot hold the whole fleet, every replacement pays a cold start at once, and if the hosts were never down, the cluster has just doubled its running workloads.