How would you structure Kafka alerting around SLOs to avoid alert fatigue — symptom-based vs cause-based alerts, escalation tiers, and what should and shouldn't page?

## The problem: alert fatigue Kafka exposes hundreds of metrics. If you page on each one nonzero, one broker failure fires dozens of simultaneous pages, on-call burns out, and real signals get ignored. The fix is to design alerting top-down from **SLOs** (what users are promised) rather than bottom-up from available metrics. ## SLOs for Kafka Map to what consumers of the platform actually experience: - **Availability**: producers and consumers can read/write — proxied by offline partitions = 0 and acceptable request error rates. - **Durability**: the cluster honors its replication/acks contract — proxied by ISR >= min.insync.replicas (no UnderMinIsr) and zero unclean leader elections. - **Freshness (for streaming consumers)**: records processed within a target latency — proxied by consumer lag-in-seconds under an SLO. Each SLO gets a target (e.g. 99.95% of minutes with zero offline partitions) and an **error budget** (the allowed shortfall). ## Symptom-based vs cause-based alerts - **Symptom-based** (page-worthy): describe a user-visible failure happening *now*. Offline partitions, UnderMinIsrPartitionCount > 0, consumer lag past the freshness SLO, ActiveControllerCount sum != 1, sustained produce/fetch error rates. These page because the SLO is breaching. - **Cause-based / diagnostic** (ticket/dashboard): explain *why* something might break — URP without a min-ISR breach, disk at 70-80%, rising controller event-queue size, GC pause time, request-handler thread-pool idle ratio dropping. These are essential for diagnosis and prevention but are not themselves user-visible failures. Rule of thumb: **page on symptoms, diagnose with causes.** A cause-based alert is promoted to a page only when it's *predictive of an imminent SLO breach* — e.g. disk fill rate projects 100% within hours (a partition going offline soon), or lag burn-rate is steep enough to blow the freshness budget within the shift. ## Escalation tiers 1. **Page (critical)** — SLO breaching or about to: offline partitions, write failures, controller absent, fast lag burn. Wakes someone. 2. **Ticket (warning)** — degraded, fix during business hours: sustained URP, disk 70-80%, slow controller queue, elevated latency, slow lag burn. 3. **Dashboard / info** — trends and capacity: throughput, partition counts, ISR churn history. No notification. ## Burn-rate / error-budget alerting Borrowed from SRE: alert on how fast you're consuming the error budget. A **fast burn** (budget gone in hours) pages; a **slow burn** (budget gone in days) tickets. This naturally separates urgent from chronic without per-metric threshold guessing, and is ideal for lag and availability. ## Noise-reduction mechanics - **Sustained windows** ('for 10m') on everything page-worthy except hard outages, so transients don't fire. - **Inhibition / grouping**: when a broker is down, suppress the 200 downstream URP/leadership alerts and raise one 'broker X down' incident. Alertmanager-style inhibition rules or a higher-severity alert that mutes its children. - **Maintenance silences** around rolling restarts and reassignments. - **Dedup by root cause**: alert on counts and the common dimension (broker, AZ), not per-partition. ## Runbook linkage and escalation policy Every page-worthy alert links to a runbook and an escalation path (on-call → data-platform owner → vendor). This is what turns a metric into an actionable incident response and keeps MTTR low. ## Edge cases / tradeoffs - Too few alerts and you miss slow-burn degradations (e.g. lag creeping toward the SLO); burn-rate alerting covers this. - Symptom-only paging can leave you blind to *why*; that's exactly why cause-based alerts still exist — just routed to tickets/dashboards. - SLO targets must be set with the consumers of the platform, not invented by ops, or the error budget is meaningless.