A CloudWatch alarm on your service's error-count metric stayed in INSUFFICIENT_DATA throughout a real outage instead of firing. Explain how a CloudWatch alarm evaluates a metric — period, evaluation periods, datapoints to alarm — and how the missing-data treatment decides what happens when the metric stops arriving.

## The five settings that define an alarm A metric alarm is not "a line on a graph". It is a small state machine over a fixed set of inputs: - **Period** — how wide each datapoint is (60 seconds, 300 seconds, and so on). This must match how the metric is actually published; asking for a 60-second period on a metric published every five minutes gives you four empty periods out of every five. - **Statistic** — how the raw observations inside a period are collapsed: `Sum`, `Average`, `Maximum`, `Minimum`, `SampleCount`, or a percentile such as `p99`. - **Threshold and comparison operator** — for example `GreaterThanOrEqualToThreshold` with a threshold of 5. - **EvaluationPeriods (N)** — how many recent periods the alarm considers. - **DatapointsToAlarm (M)** — how many of those N must breach. When you omit it, M equals N. The M-out-of-N form exists to trade responsiveness against flapping. `3 out of 3` needs a sustained condition and reacts slowly; `2 out of 5` tolerates a single spiky period while still catching a persistent problem within roughly five periods. ## The three states An alarm is in `OK`, `ALARM`, or `INSUFFICIENT_DATA`. Actions — an SNS notification, a scaling action — are attached to **transitions** into a state, not to being in it. An alarm that has been in ALARM for six hours has sent exactly one notification. That surprises people who expect a repeat page, and it is why re-notification is a job for whatever consumes the SNS topic. ## What actually happens when data is missing CloudWatch first tries hard to find datapoints. If some of the last N periods are empty, it extends the window backwards — the *evaluation range* — collecting up to N real datapoints from a longer stretch of time before it gives up. Only genuinely absent data reaches the `TreatMissingData` setting, which has four values: | Setting | A missing period is scored as | Effect | |---|---|---| | `missing` (default) | not evaluated | Alarm keeps its current state; goes INSUFFICIENT_DATA when the whole range is empty | | `notBreaching` | good | Gaps push the alarm towards OK | | `breaching` | bad | Gaps push the alarm towards ALARM | | `ignore` | — | State is frozen; the alarm never changes on missing data | The failure in the question is the default at work. The alarm watched `Sum` of an error metric. When the service died it stopped publishing anything, including errors. Every period became missing, the evaluation range emptied, and the alarm slid into INSUFFICIENT_DATA — technically correct, operationally useless. ## Three ways to fix it **Score gaps as breaching.** If the metric is genuinely published on every period the service is healthy, `--treat-missing-data breaching` makes silence itself the alarm condition. Use it only when you are sure of the publication cadence, or a deployment restart will page you. **Alarm on a metric that is always present.** AWS-vended metrics differ here: an SQS queue publishes `ApproximateAgeOfOldestMessage` every minute whether or not anything is happening, whereas Lambda publishes `Errors` only when invocations occurred. Prefer the continuously-published signal, or watch a positive signal (request count, successful heartbeat) with a `LessThanThreshold` comparison, which fires when the number drops rather than when a rare number appears. **Fill the gap in metric math.** An expression such as `FILL(m1, 0)` substitutes zero for missing periods, converting a gap into a real datapoint before the alarm sees it. That is the right tool when you want zero to mean zero — but note it also means an alarm on `FILL(errors, 0)` will never fire from silence, which may be the opposite of what you want. ```bash aws cloudwatch put-metric-alarm \ --alarm-name checkout-errors \ --namespace MyApp --metric-name Errors \ --statistic Sum --period 60 \ --evaluation-periods 5 --datapoints-to-alarm 3 \ --threshold 5 --comparison-operator GreaterThanOrEqualToThreshold \ --treat-missing-data breaching \ --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-pager ``` ## Two more sharp edges An alarm on a **percentile** statistic has an extra setting, `EvaluateLowSampleCountPercentile`. Set it to `ignore` and periods with too few samples do not move the alarm — a single slow request in a quiet minute will not page you. And an alarm evaluates on a delay: the metric has to be published, aggregated and made available before the period can be judged, so the practical detection time is roughly `period × N` plus a minute or two. Choosing a five-minute period with three evaluation periods buys you a sixteen-minute worst case before anyone is told.