Observability & Monitoring

A Prometheus alerting rule pairs a PromQL expression with an optional for duration, plus labels and annotations. The for duration requires the expression to hold continuously across that window, so the alert sits in pending before it becomes firing.

One line carries a metric name, an optional label set in braces, a float64 value and an optional millisecond timestamp. The HELP comment describes the family and the TYPE comment declares counter, gauge, histogram, summary or untyped.

An instant vector selector returns one sample per series at the query timestamp; a range vector selector adds a bracketed duration and returns every sample in that window. Functions like rate() only accept the bracketed range form.

Each cycle Prometheus sends one HTTP GET to the target's metrics path, parses the body, and stores every sample under that scrape's start timestamp. scrape_interval sets how often it repeats; scrape_timeout bounds one request. Prometheus also records an up sample per scrape.

Prometheus builds its target list from the discovery blocks in each scrape job: a static list, files on disk, or a platform API such as Kubernetes or EC2. Each target arrives carrying read-only metadata labels describing its origin.

Time series plots every point over time. Stat and Gauge reduce each series to one number (last, mean, max); Gauge adds a bounded min/max scale. Table shows the raw rows. Heatmap shows distribution. Choose by trend vs single value vs distribution.

A data source is a saved connection (type, URL, auth, options) plus the plugin that knows how to query that system. The browser asks the Grafana server, which attaches the stored credentials, calls the backing system, and returns typed data frames the panel renders.

Use a Query variable: Grafana runs a metadata query against the data source (e.g. a label-values lookup) and turns each returned row into a drop-down option. Refresh decides when that query re-runs — never (options are read from saved dashboard JSON), on every dashboard load, or additionally whenever the time range changes.

Each query runs and returns labelled series. Server-side expressions then chain on those results — reduce a series to one number, do math, apply a threshold — and one expression is marked the rule's condition. Every distinct label set that satisfies it becomes its own alert instance.

An instance is Normal until the condition breaches, then Pending until it has breached continuously for the pending period, then Alerting, which notifies. Missing data and query failures are separate states whose handling is configurable per rule: treat as alerting, as normal, as a dedicated no-data alert, or keep the last state.

A Zabbix item is one configured metric on a host, named by a key. In a passive check the Zabbix server connects to the agent and asks for a value; in an active check the agent connects out and sends.

A Zabbix trigger is a boolean expression over an item's stored history, built from functions like min or avg over a window rather than a threshold on the latest value. A separate recovery expression clears the problem on a looser condition.

A Zabbix template is a reusable container of items, triggers and discovery rules linked to hosts; editing it changes every linked host at once. Low-level discovery expands prototypes into one item or trigger per entity found on each host.

A Zabbix server is sized by new values per second, not host count. Collection saturates first: pollers are a fixed pool, so checks are delayed and queue up. A proxy moves collection off the server and reaches networks it cannot address.

The Datadog Agent is one host process with several separate intake paths: it collects host metrics itself, runs integration checks that poll local services, accepts custom metrics pushed to its DogStatsD listener, and receives spans from in-process tracing libraries.

Every signal must carry the same env, service and version tags, set on the Agent, the tracing library and the log records. Log records must also carry the active trace and span identifiers to be joined to one request.

A Datadog monitor only reaches a person if its notification message addresses a recipient handle; without one it records state silently. Storms are controlled by deliberate query grouping, composite monitors, and scheduled downtimes.

Attack the two billed units directly: custom metrics, counted per unique metric-name-and-tag-value combination, and indexed logs, billed separately from ingestion. Bound what tags may contain, aggregate before submitting, and decide after ingest which logs are worth indexing.

New Relic writes every signal - events, metrics, logs and spans - as timestamped, attributed records in one store, NRDB, and reads them all with one language, NRQL. Correlating a log with a span becomes a query, not an export.

NRQL aggregates records at query time: SELECT chooses the aggregation, FROM the data type, WHERE filters, FACET splits results by an attribute's values, SINCE and UNTIL set the window, and TIMESERIES cuts that window into buckets.

New Relic attributes time only to the spans it received. Work in an uninstrumented hop shows up as unexplained time inside its caller's span, and the platform cannot say whether that was network, queueing, or the missing service's own work.

An entity in New Relic is anything the platform monitors and can name - a service, host, container, database or browser app - identified by an entity GUID. Incoming telemetry is matched to one by the identifying attributes it carries.

Dynatrace's OneAgent installs once per host, discovers the processes running there, and injects instrumentation into supported runtimes so traces, code-level timings and host metrics appear with no code change. It cannot supply business meaning or cover unsupported runtimes.

Dynatrace's agents continuously report hosts, processes, containers and the connections between them, and the platform keeps that as a time-versioned entity graph. Automated root cause needs it because causation is a walk over dependency edges, not a correlation of charts.

A Dynatrace verdict is a ranked hypothesis, not a diagnosis. From dependency edges, timing and change events it can say where a degradation started and what sits downstream. It stays blind to correctness bugs and causes outside the monitored estate.

The Collector is a separate process that receives, processes and re-exports telemetry. It decouples apps from backends: batching, retry, enrichment, redaction, translation and routing become config in one place instead of code in every service.

traceparent is four hyphen-separated fields: version, 32-hex trace id, 16-hex span id of the caller, and 2-hex flags whose lowest bit means sampled. The receiver continues the same trace id, parents its span on that span id, and forwards tracestate unchanged apart from its own entry.

A span carries identity (trace id, span id, parent span id), a name, a kind, start and end timestamps, attributes, events, links and a status. Kind tells the backend the span's role in a call (server, client, producer, consumer, internal); status is Unset, Ok or Error.

Zero-code agents patch libraries at load time and emit spans for inbound requests, outbound calls, database and messaging clients with no source changes. Code-based instrumentation adds spans and attributes only your code knows. Both feed one SDK and nest.

OTLP is the one exporter every OpenTelemetry SDK ships and it carries traces, metrics and logs, so the destination stays a config change. A native exporter compiles a vendor's protocol into the process, making a backend switch a redeploy. Use gRPC in-cluster, HTTP where proxies or browsers break it.

A span leaves the instrumented process through a bounded in-process reporter queue, reaches a Jaeger collector, and is written to storage, optionally via Kafka and an ingester. Every hop buffers, and every buffer drops spans when it fills.

Compare a span's duration with the time its children cover. A parent's bar encloses its children, so the slow hop is the one with large self time — duration minus child coverage — not simply the longest bar on the screen.

Jaeger clients poll the backend for their sampling rate, so per-service and per-operation rates change without a redeploy. Central configuration cannot pick traces by outcome, cannot bound spans per trace, and does nothing for a client that never fetches it.

A Jaeger store must absorb write-heavy append-only ingest, answer point lookups by trace identifier, and serve ad-hoc search by service, operation, tag and duration. Cassandra needs extra index tables for that search; a search engine indexes everything at ingest instead.

A Sentry SDK sends the exception type, message and stack trace plus the surrounding scope: request, user, tags, release and a trail of breadcrumbs. Minified or compiled frames stay unreadable until matching source maps or symbol files are uploaded.

A release stamps events with the deployed version, so Sentry shows which deploy first produced an issue, marks a reappearance as a regression, and points at suspect commits. Resolving in the next release reopens the issue only if it recurs afterwards.

Sentry groups events by a fingerprint computed from the event itself -- normally the stack trace's in-app frames and the exception type, falling back to the exception value or message when there is no trace. Same fingerprint, same issue.

Volume is cut in the SDK -- an error sample rate, ignore lists, a before-send hook that drops the event -- and at ingest by inbound filters, client-key rate limits and spike protection. Alert on new or regressed issues, not event counts.

Parse where you can afford the CPU. Filebeat ships cheaply on the host; a Logstash tier adds heavy enrichment and multi-destination routing but costs a tier to run; an Elasticsearch ingest pipeline parses inside the cluster.

Inputs receive events, filters transform them, outputs write them, in that order. A grok pattern that fails to match does not drop the event: Logstash tags it _grokparsefailure and ships it unparsed, after burning more CPU than a successful match.

Agree a common vocabulary such as the Elastic Common Schema and enforce it in the shared shipping tier, not in documentation. When two services disagree on a field's type, one index rejects documents and searches across several go quietly wrong.

Logstash's persistent queue writes in-flight events to disk before filtering, so a restart or a downstream outage does not lose them. The dead-letter queue holds events Elasticsearch will never accept. Backpressure fills the queue and stalls Filebeat.

A Kibana data view names the indices a search may touch, usually a wildcard, and exposes their merged field list to Discover and dashboards. Nominating a time field is what makes the global time picker filter queries built on it.

A KQL query is one text expression for the whole search; a filter pill is a discrete condition you can negate, disable or pin. KQL has no regex, fuzziness or boosting; for those, switch the bar to Lucene syntax.

A Kibana dashboard is a saved object that points at other saved objects by id: its panels and the data views behind them. Export it without those references and it imports cleanly but renders nothing.

Loki indexes only the label set attached to each log line, never the line's text. A stream is one unique combination of label values; its lines are batched into compressed chunks kept in object storage.

Each distinct label value creates a separate Loki stream with its own chunks. A high-cardinality label multiplies streams, so the index swells, ingesters hold thousands of barely-filled chunks in memory, and object storage fills with tiny, poorly-compressed files.

The stream selector decides which chunks are read from object storage. Every filter, parser and metric stage after it runs over lines already fetched and decompressed, so they change what you see, not how much is read.

The agent picks the stream labels through discovery and relabelling, joins continuation lines into one entry, and sets each entry's timestamp. All three are expensive to revisit: labels and timestamps are baked into chunks that are immutable once flushed.

A Graylog input binds a protocol to a port and its codec decodes each payload into a message with source, message and timestamp fields. Extractors on that input parse further; pipeline rules do so later, across inputs.

Every Graylog message is tested against all stream rules and can join several streams, each with its own index set. Connected pipelines then run stage by stage and may enrich, re-route or drop it. Unmatched messages stay in the default stream.

Route that team's messages into their own Graylog stream, grant read on just that stream through a role or an entity share, and define event definitions whose search is scoped to it. Alerting and access both hang off the stream boundary.

At index time Splunk fixes an event's index, timestamp, host, source and sourcetype. Fields, aliases and lookups are worked out at search time and can be changed at any moment. The index-time decisions stick: correcting one means re-ingesting the data.

An SPL search starts with a retrieving stage that pulls events from Splunk indexes over a time range, then pipes them onward. Streaming stages act on one event at a time; transforming stages must gather the whole result set first.

A universal forwarder is a light agent shipping a host's raw data. A heavy forwarder parses events first, so it can mask, drop and route them before indexing. A network input accepts pushed data with no agent and no source-side buffer.

A summary index stores a scheduled aggregation's results as small events you search instead of raw data; report acceleration has Splunk maintain equivalent precomputed data for a qualifying saved search. Both buy speed with freshness, storage and recurring load.

Log levels are a contract about who must act. ERROR means something failed and a human should look; WARN means something recovered but degraded; INFO records state changes; DEBUG and TRACE are developer detail, off by default in production.

A full-text index built at ingest makes any word in any line searchable without declaring it first, and makes aggregations fast. You pay twice: CPU on the write path for every record, and index bytes stored alongside the logs.

Structured logging emits each line as named, typed fields instead of one formatted sentence. Ingest no longer has to guess where each value starts and ends, and queries can filter, compare and aggregate on a field rather than searching raw text.

A time series is the metric name plus its complete label set; change any label value and it is a different series. Adding a label multiplies the series count by that label's distinct values rather than adding to it.

A counter only ever increases and returns to zero when its publishing process restarts; a gauge is a level that moves either way. Because a counter's absolute value depends on process uptime, you read its rate over a window.