An EC2 instance shows "1/2 checks passed" in the console. Explain the difference between the system status check and the instance status check, and how your response differs depending on which one failed.

## The checks EC2 runs for you Every EC2 instance is probed roughly once a minute by checks that pass or fail independently. The console summarises them as "2/2 checks passed"; the interesting states are the partial ones, because which half failed tells you whose problem it is. - **System status check** — the infrastructure the instance sits on: host hardware, host software, loss of power, loss of network connectivity to the host. A failure here means AWS's side is impaired. - **Instance status check** — the instance's own configuration and guest operating system: it verifies that the instance is reachable and responsive at the OS level. - **Attached EBS status check** — a separate check reporting whether the instance's attached EBS volumes are reachable and able to complete I/O. The corresponding CloudWatch metrics in the `AWS/EC2` namespace are `StatusCheckFailed_System`, `StatusCheckFailed_Instance`, `StatusCheckFailed_AttachedEBS`, and the aggregate `StatusCheckFailed`. Each publishes 0 for pass and 1 for fail. ## System check failed: move the instance Because the fault is in the host, nothing you do inside the guest changes it. The correct action is to get onto different hardware: - **Stop and start** the instance. This releases it from the impaired host and schedules it onto a new one. Everything on EBS survives; instance-store data does not, and the auto-assigned public IPv4 address changes. - **Wait**, if the impairment is a transient host issue AWS is already remediating. Scheduled maintenance and instance retirement notices show up as AWS-scheduled events on the instance. The trap here is the reflex to reboot. `RebootInstances` restarts the guest **on the same host**, so it does exactly nothing for a system-level impairment. Distinguishing reboot from stop/start is the single most common discriminator in this question. ## Automatic recovery EC2 provides automatic recovery for a system-impaired instance: it is migrated onto new hardware while keeping its instance ID, private IPv4 address, any Elastic IP, its EBS volumes and its metadata. Modern instance types have simplified automatic recovery enabled by default, and you can also drive it explicitly with a CloudWatch alarm action: ```bash aws cloudwatch put-metric-alarm --alarm-name ec2-system-check-recover \ --namespace AWS/EC2 --metric-name StatusCheckFailed_System \ --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \ --statistic Maximum --period 60 --evaluation-periods 2 --threshold 1 \ --comparison-operator GreaterThanOrEqualToThreshold \ --alarm-actions arn:aws:automate:us-east-1:ec2:recover ``` The constraint worth remembering: recovery is not available for instances that use instance-store volumes, because there is no way to move those disks. That fact quietly shapes the design — a workload that depends on instance store cannot be auto-recovered and must be rebuildable instead. ## Instance check failed: fix the guest A failed instance check is on your side of the line. Typical causes: a kernel panic, a full root filesystem, a corrupted file system that dropped the instance into a repair prompt, exhausted memory, a misconfigured network interface or firewall rule inside the OS, or a boot that stalled on a bad `/etc/fstab` entry after a volume was detached. The sequence is: reboot first (it clears transient states and is cheap), and if the check does not clear, look at what the instance printed. `GetConsoleOutput` returns the serial console buffer — kernel messages, boot output, panic traces — and `GetConsoleScreenshot` gives you the actual screen, which is how you catch a machine sitting at a filesystem-repair prompt. The EC2 Serial Console gives interactive access to a machine whose network stack is broken, and it works when SSH does not, which is precisely when you need it. As a last resort, stop the instance, detach the root volume, attach it to a healthy instance to repair the filesystem or configuration, and reattach. ## Alarming on it Status checks are the cheapest health signal EC2 gives you and they are per-instance rather than per-application: they will tell you the box is broken and never that the application is. Alarm on `StatusCheckFailed_System` with a recovery action, and on `StatusCheckFailed_Instance` to page or to trigger replacement. For instances in an Auto Scaling group, the group's health check already acts on these signals and replaces the instance, which is generally a better answer than recovering it — a fleet member should be replaced, not repaired. ## The judgment an interviewer wants The strong answer separates three responses: *reboot* fixes a guest problem, *stop/start or recovery* fixes a host problem, and *replace* is what you do when the instance is a fungible member of a fleet. The weak answer reboots everything and hopes. Say out loud that a repeated instance-check failure across replacements is an AMI or bootstrap defect and not an infrastructure event — that is the point at which you stop operating and start fixing the image.