As the platform lead for an engineering organisation, how do you decide which workloads may run on EC2 Spot and which must never, and how do you bound the damage when Spot capacity disappears across the fleet?

## Turn the question into a property of the workload The wrong version of this conversation is a negotiation between a finance-minded platform team pushing Spot and product teams resisting it. The right version is a **classification** that every workload gets, applied consistently, with the tradeoff written down. The classifying question is narrow and answerable: *if this instance disappears in two minutes, with no human involved, what is the cost?* Break it into four sub-questions: 1. **Is the work retryable without human intervention?** If a killed unit of work simply reappears and completes elsewhere, the eviction cost is near zero. 2. **Is any state on that node the only copy?** In-memory session state, an unreplicated database volume, an accumulated cache that takes an hour to rebuild — each converts an eviction into a real loss. 3. **Is the effect customer-visible?** Latency, a failed request, a missed deadline. Elastic capacity absorbing a traffic peak fails differently from the last instance serving a checkout page. 4. **What is the restart cost relative to the discount?** A job with a twenty-minute warm-up that is interrupted every hour has negative savings, no matter what the hourly rate says. ## The categories that fall out **Spot by default.** CI and CD build agents, test environments, batch and ETL with checkpointing, media transcoding, model batch inference, load generators, big-data worker nodes, stateless queue consumers. These already treat a node as disposable, and the discount is close to free money. **Spot for the elastic tier only.** Web and API fleets, stream processors, container clusters. Here the honest position is not "Spot or not" but *how much*: a floor of On-Demand capacity sized to the traffic that must always be served, with Spot carrying the peaks. In an Auto Scaling group's mixed-instances policy that is `OnDemandBaseCapacity` for the floor and `OnDemandPercentageAboveBaseCapacity` for the split above it. **Never Spot.** Self-managed database primaries and any singleton holding the only copy of state; message brokers with undelivered messages on local storage; licence and identity servers everything depends on; cluster control-plane or coordinator nodes where quorum loss is catastrophic; long non-resumable jobs; and anything on a contractual deadline whose miss costs more than a year of the savings. Publish this list — it is far more useful as a short, explicit prohibition than as case-by-case judgment. ## Bounding the blast radius Suitability decides *whether*. These decide *how badly it can go*. - **An On-Demand floor per fleet**, sized to committed throughput rather than to a comfortable-sounding percentage. That floor is the guarantee, bought deliberately at full rate. - **Pool diversification as a platform default.** A fleet allowed to use only one instance type in one zone has correlated failure regardless of how suitable the workload is. Many interchangeable types across all usable zones, with a capacity-aware allocation strategy, is a default the platform sets — not a per-team decision. - **Graceful interruption handling in the shared runtime.** A base image or sidecar that watches for the interruption notice and drains, checkpoints and exits cleanly, so every team inherits correct behaviour instead of writing it badly. - **Correlation awareness across teams.** If every fleet in the organisation diversifies into the same three popular instance types, the organisation is still concentrated. Spread the *portfolio* across families and generations, not just each fleet. - **A fallback rule.** Decide in advance what happens when Spot capacity is unavailable for an extended period: automatically top up with On-Demand and eat the cost, or let the queue grow. Both are defensible; discovering which one you chose during an incident is not. ## Measure the savings honestly The discount is gross. The net is the discount minus retried work, minus engineering time spent on interruption handling, minus incidents. Instrument it: record every interruption with instance type and zone, chart interruption rate per pool, and track wasted instance-hours from runs that never completed. That data ends most arguments — it identifies pools to stop using, and it shows when a fleet's Spot ratio has been pushed past the point where it still pays. ## The organisational trap to name The subtle failure is **Spot savings silently becoming an availability promise**. A team runs at 90% Spot, the interruption rate happens to be low for a quarter, the savings get booked into a budget, and the fleet is now implicitly committed to capacity nobody guaranteed. When the pool tightens, the choice is a visible outage or an unbudgeted cost — and that decision arrives during an incident, at the worst possible time. The defence is to keep the guarantee and the discount separate in the way the organisation talks about them: reliability comes from the On-Demand floor and from diversification, and Spot is an *upside* on top. If a service's stated availability depends on Spot capacity being there, the stated availability is wrong, and that is a conversation to have on a calm afternoon rather than at 3 a.m.