In Zabbix, what does a trigger expression evaluate, and what does a recovery expression add?

## A trigger is an expression, not a threshold A Zabbix **item** collects values; it never decides anything. The decision lives in a **trigger**: a named boolean expression attached to one or more items, with two states, `OK` and `PROBLEM`. Zabbix re-evaluates a trigger when a new value arrives for an item it references, and expressions built on time-based functions are re-evaluated periodically as well, so they can fire when nothing arrives at all. The expression is not "value greater than number". It is built from **trigger functions**, each of which reads a slice of the item's stored history: ``` min(/ferry-booking-web-03/system.cpu.util,5m)>90 ``` Read it left to right: take the item `system.cpu.util` on host `ferry-booking-web-03`, look at every value collected in the last five minutes, take the smallest, and compare it with 90. That trigger opens only if utilisation stayed above 90% for the *whole* window, so a single spike cannot raise it. Swap `min` for `avg` and you have made a weaker statement; swap it for `last` and you are back to a bare threshold on the newest value. The common functions are worth knowing by name: - `last()` — the most recent value; the only one that behaves like a naive threshold. - `min()`, `max()`, `avg()` over a period — the sustained-condition family. - `count()` — how many values in the period matched a condition. - `change()` — the difference between the last two values. - `nodata()` — true when *no* value has arrived for the period. This is how a silent host or a stalled collection path is detected, and nothing based on the latest value can express it, because there is no latest value. Functions take either a time period (`5m`, `1h`) or a count of values (`#3` for the last three), and one expression may reference items on several hosts, which is how "the queue is deep *and* the database is lagging" becomes a single problem rather than two correlated ones. ## What a recovery expression buys you By default a trigger returns to `OK` when its problem expression stops being true — the setting Zabbix calls **OK event generation: Expression**. Change it to **Recovery expression** and you supply a second, independent expression that must become true before the problem closes. The reason is **hysteresis**. Consider a metric parked near its limit: CPU on the ferry-timetable booking API sitting between 89% and 92% through the evening sailing rush. With one expression at `>90` the trigger opens and closes on every crossing — forty-one state changes in an hour, forty-one problems, and a problem history nobody can read. Raising the threshold does not help; it just moves the line the metric oscillates around. A recovery expression separates the two edges: | | Single expression | Problem plus recovery expression | |---|---|---| | Opens when | `min(...,5m)>90` | `min(...,5m)>90` | | Closes when | the same expression is false | `max(...,10m)<75` | | Behaviour near the limit | flaps on every crossing | one problem, held until clearly recovered | | Duration reported | fragmented into slivers | the real duration of the incident | | Failure mode | alert fatigue | a problem open slightly longer than the fault | The gap between 90 and 75 is a **deadband**: while the metric is inside it, the trigger keeps whatever state it already has. Note that the recovery expression is deliberately *not* the negation of the problem expression — a candidate who writes `max(...,5m)<=90` as the recovery has re-created exactly the flapping they set out to remove. The same mechanism covers cases where recovery is a genuinely different measurement rather than a looser bound: open on `nodata()` for an agent availability item and close on the first successful value; open when a queue depth exceeds a limit and close only after it has been drained for a sustained window. ## Getting the shape right A handful of choices decide whether a trigger is usable in production: 1. **Choose the function before the number.** Most bad triggers are `last()` with a carefully tuned threshold where `min()` or `avg()` over a window was wanted. 2. **Set severity honestly.** Zabbix offers Not classified, Information, Warning, Average, High and Disaster. Severity is a property of the trigger, and everything downstream filters on it. 3. **Use trigger dependencies.** When a host is unreachable, its twenty per-service triggers are noise; making them depend on the host-unavailable trigger suppresses the derived problems while the root problem is open. 4. **Reference items, never duplicate them.** A trigger stores nothing of its own, so an expression whose window is longer than the referenced item's history retention silently evaluates over less data than you think. 5. **Watch the evaluation cost.** A long window over a high-frequency item is recalculated on every arriving value; hundreds of those across a fleet is real server work, not a rounding error.