Instances in an EC2 Auto Scaling group pass their EC2 status checks, but the application on them returns errors and the load balancer has marked them unhealthy. Why has Auto Scaling not replaced them, and what would you change?

## Two independent opinions about health There are two systems forming a view of these instances, and by default they do not talk to each other. **EC2 status checks** are the platform's own. The *system status check* covers the underlying host and network — a failure means AWS infrastructure is broken and usually requires a stop/start onto new hardware. The *instance status check* covers the virtual machine: has it booted, is its network stack responding. Neither knows anything about your process, your port, or your dependencies. A machine sitting at a login prompt with your service crashed passes both. **The load balancer's health check** probes the application through its target group and is the thing that decides whether traffic is sent. When it fails, the target is drained. ## What HealthCheckType actually selects The group's `HealthCheckType` decides which of those opinions can cause a *replacement*: - `EC2` (the default) — platform status checks only. - `ELB` — platform status checks **plus** the health status reported by the attached load balancer target groups. `ELB` is additive, not a replacement: a failed system status check still gets the instance replaced. So the described symptom is exactly the default behaviour. The load balancer stops sending traffic to the broken instances, the group counts them as healthy and keeps desired capacity satisfied with them, and available serving capacity silently collapses while the fleet size looks correct on every dashboard. The classic version of this incident is a fleet of ten instances where eight are drained and two are absorbing all production traffic until they fall over too. ```bash aws autoscaling update-auto-scaling-group \ --auto-scaling-group-name web-asg \ --health-check-type ELB \ --health-check-grace-period 300 ``` ## The grace period is not optional detail `HealthCheckGracePeriod` is the window after an instance launches during which health checks cannot mark it unhealthy — it exists because a booting instance legitimately fails an application probe for a while. Its default is 300 seconds as of 2025. If your application needs longer than the grace period to become ready, turning on `ELB` health checks creates a **launch loop**: the instance boots, is judged unhealthy before it is ready, is terminated, and a replacement starts the same journey. You burn money, churn the fleet, and never reach capacity. When someone reports "my group keeps replacing instances every few minutes", an undersized grace period is the first thing to check — and the honest fix is usually to measure real time-to-ready and set the grace period above the worst case, not to disable the health check. ## Replacement order and available capacity By default the group terminates an unhealthy instance and then launches its replacement, meaning capacity dips during the swap. An instance maintenance policy lets you express the tolerance explicitly with a minimum and maximum healthy percentage, so you can require the replacement to be launched *before* the old one goes away. On a small group that difference is significant: replacing one instance out of three is a third of your capacity gone for the length of a boot. ## The judgment part The technically correct answer is `HealthCheckType ELB`. The senior answer adds a caveat, because automatic replacement is only the right response to an *instance-specific* fault. Ask what all the failing instances have in common. If they are failing because a shared dependency is down, or because the version they are running is broken, then replacement is actively harmful: each new instance inherits the same defect, fails, and is replaced again. You get a churn loop that consumes launch capacity, floods your logs with fresh instance ids that make correlation harder, and — because instances are being terminated mid-boot — can leave you with less healthy capacity than if you had done nothing. The signature is that instance ids keep changing while the error rate does not. In that situation the levers are to suspend the `ReplaceUnhealthy` and `HealthCheck` process on the group while you diagnose, or to roll back what is being launched. Suspending is a deliberate, temporary act with an owner and a plan to resume — not a way of making a red dashboard turn green. ## What sits underneath One boundary worth being explicit about in an interview: the *content* of the probe — which path, how many consecutive failures, what timeout — is configuration on the target group, not on the Auto Scaling group. The group only consumes the resulting verdict. Getting that separation right in your answer signals that you understand the layering rather than having memorised a checkbox.