Observability foundations
The tool-agnostic ideas every monitoring stack is built on: telemetry signals, metric math, log pipelines, and tracing with sampling. Interviewers use these to check whether you understand telemetry itself or only one vendor's UI.
on this pageshowhide
guide
overview
~1 minObservability interviews usually start below the tools. Before anyone asks about a particular collector, query language or dashboard, the interviewer wants to know whether you understand the telemetry itself: what kind of data each signal is, what it costs to keep, and which questions it can and cannot answer. A candidate who knows one product's screens but cannot explain why a percentile does not average, or why a trace falls apart when one service forgets to forward its context, is found out quickly. The ideas hold from one service to a large fleet, so they appear at every level. The subject has four sections. [Telemetry signals](/topics/cloud-observability-foundations-signals) compares metrics, logs, traces, wide events and profiles, and asks which one you reach for first. [Metrics concepts](/topics/cloud-observability-foundations-metrics-concepts) covers metric types, labels and cardinality, histogram math, push versus pull collection, and the RED and USE checklists. [Logging concepts](/topics/cloud-observability-foundations-logging-concepts) treats logging as a system: structured records, severity levels, the aggregation pipeline, storage models and the bill. [Tracing and sampling](/topics/cloud-observability-foundations-tracing-sampling) covers spans and context propagation, correlation between signals, head and tail sampling, and how instrumentation gets into a service in the first place. Start with the signals section, because every later answer assumes you can say what a given signal keeps and what it throws away. Metrics come next, since cardinality and percentile math are where confident answers most often go wrong; then logging and tracing, which share the problem of carrying request context across process and thread boundaries. Senior and principal questions rarely introduce a new mechanism. They ask where a control should sit, what it costs, who pays for it, and what a decision made today makes impossible later.
primer
A few ideas run through every section. Hold these and most of the questions below become applications of them rather than separate facts. - **Every signal is a decision about what to throw away.** A metric keeps counts and discards identity. A log line keeps one event's detail and discards the causal links between services. A trace keeps one request's path and, once sampled, discards most requests. A question about which signal to use is really a question about which loss the investigation can tolerate. - **Cost follows shape.** Metric cost grows with the number of distinct series; log and trace cost grows with traffic. That is why the remedies differ: you bound labels for metrics, and you filter, sample or tier for logs and traces. Naming the axis that drives a bill is most of a cost discussion. - **Aggregates do not compose freely.** A counter's raw value depends on restarts, so it is read as a rate. Percentiles and ratios from separate processes cannot be averaged into a fleet figure. A histogram's precision is fixed by bucket boundaries chosen before the first measurement. Much of the metric math in interviews tests whether you know which quantities add and which do not. - **Cardinality multiplies.** A series is a name plus its full label set, so each new label multiplies how many series exist instead of adding a few. The dangerous label is the one whose value set has no ceiling, not the one that happens to be large this week. - **Context has to be carried; it is never there by default.** A trace exists only if every hop forwards its identifiers. A log line joins its request only if that request's identity reaches the logger. A sampling decision stays coherent only if downstream services inherit it. Thread pools, queues and asynchronous hand-offs are where context goes missing, and interviewers lean on exactly those cases. - **Early decisions are the permanent ones.** Data you never recorded, never collected or discarded before storage cannot be recovered, and a metric type or log schema chosen at write time binds everything written under it. Later stages can reshape or delete but not restore. When an early decision is unavoidable, make it explicit and count what it drops. - **Monitoring and observability are complementary.** Monitoring checks conditions someone predicted; observability lets you ask new questions of data kept at enough detail. Design conversations are usually about how much to spend on each.
- Time series
- A sequence of timestamped values identified by a metric name together with one complete set of label values.
- Label
- A key-value dimension attached to a metric, also called a tag or attribute, that lets one measurement be split and filtered.
- Cardinality
- The number of distinct values a label takes, or the number of distinct series a metric produces; the main driver of metrics storage cost.
- Counter
- A metric that only increases during a process's lifetime and resets when the process restarts; normally read as a rate over a time window.
- Gauge
- A metric reporting a current level that can rise or fall, such as queue depth or memory in use.
- Histogram
- A metric that counts observations into predefined value ranges, so distributions and approximate percentiles can be computed and merged across instances.
- Percentile
- The value below which a given share of observations falls; a p99 latency is the time that 99 percent of requests stayed under.
- Span
- One timed operation within a trace, carrying its own identifier, its parent's identifier and attributes describing the work done.
- Context propagation
- Passing trace identifiers and the sampling decision from caller to callee, across network calls and in-process hand-offs, so separate spans join one trace.
- Head-based sampling
- Deciding whether to keep a trace when it begins, before its outcome is known.
- Tail-based sampling
- Deciding whether to keep a trace after its spans have arrived, so the decision can take errors and latency into account.
- Exemplar
- A reference stored with an aggregated metric value that points to one concrete trace which contributed to it.
- Structured logging
- Emitting log records as named fields rather than formatted text, so they can be filtered and aggregated without guessing at the format.
- Wide event
- One record per unit of work carrying many attributes, kept unaggregated so it can later be sliced along any of them.
- Sampling profiler
- A tool that periodically captures call stacks and aggregates them to show which code paths consume CPU, memory or another resource.
- Saturation
- Demand a resource has accepted but cannot serve yet, visible as queued work, waiting time or rejected requests.
The four sections describe one system from different sides, and most interview scenarios cross at least two of them. **Signals frame the rest.** The [telemetry signals](/topics/cloud-observability-foundations-signals) section sets out the vocabulary of loss and cost that the other three apply. Metrics, logging and tracing each take a deeper look at one signal; wide events and profiles sit beside them as answers to questions none of the three handles well: slicing per-request records along arbitrary dimensions, and attributing a resource to code rather than to requests. **Metrics are where an investigation usually begins.** Alerts and dashboards are built on aggregates because they are cheap to keep and fast to query, and the [RED and USE checklists](/topics/cloud-observability-foundations-metrics-concepts-red-use) decide which aggregates a component should expose from day one. An aggregate tells you that something changed and roughly where, not which requests were affected. **Correlation bridges the sections.** [Correlation IDs and exemplars](/topics/cloud-observability-foundations-tracing-sampling-correlation) lead from a single metric value to a trace, and from a trace to its log lines. Each link has to be emitted on purpose, and each depends on a decision made in another section: an exemplar helps only if the trace it names survived sampling, and a trace identifier in a log record helps only while that log is still retained. Sampling policy and log retention therefore belong in the same conversation even though they sit in different sections. **Instrumentation feeds everything.** Whether telemetry comes from hand-written code, an in-process agent, a proxy or the kernel decides which attributes exist at all, and therefore what any later pipeline stage, query or alert can work with. The cost of switching instrumentation accrues mostly downstream, in the dashboards and rules that depend on attribute names.
- Telemetry Signals →
Every later answer assumes you can say what each signal keeps, what it discards and which one to open first.
- Counter, Gauge, Histogram, Summary →
Counter, gauge and histogram semantics decide which arithmetic is legal on a metric; the rest of the metrics section builds on them.
- Labels and Cardinality →
Cardinality is the most common way a metrics system fails in production and the most common follow-up question.
- Structured Logging →
Fields instead of prose is the precondition for every pipeline, storage and cost decision in the logging section.
- Spans, Traces, and Context Propagation →
Spans and propagation are the model that correlation, sampling and instrumentation questions all assume.
- Head vs Tail Sampling →
Sampling is where tracing meets cost; take it last, once you know what a complete trace needs.
Averaging per-replica p99 values into a fleet figure; percentiles and ratios do not combine, so merge the raw counts and compute once.
Judging a label safe because it has few distinct values today; user identifiers, raw paths and error text have no ceiling and accumulate across retention.
Letting each service make its own sampling decision, which stores trace fragments instead of whole traces; the first service's verdict has to travel with the request.
Treating shorter retention as the cure for a log bill when ingest volume and write-time indexing are often the larger share.
Losing trace and logging context when work moves to a thread pool, queue or callback, then blaming the tracing backend for broken trees.
Logging at ERROR for conditions nobody needs to act on, which teaches on-call engineers to ignore the level that matters.
Treating utilization as the overload signal; it tops out at full while queue depth and waiting time keep climbing.
Answering one customer's complaint from a dashboard curve instead of from records that carry that request's identity.
The same handful of choices recurs across all four sections; naming the one you are making, and what decides it, is usually worth more than naming a tool. - **Detail versus cost.** More labels, more log fields and a higher kept fraction of traces all answer more questions and all cost more. The useful test is which past incidents the extra detail would actually have explained. - **Early versus late processing.** Parsing, filtering and sampling at the source are cheapest and least reversible; doing the same work centrally or at query time keeps options open and moves the cost to the pipeline or to every reader. - **Deciding blind versus deciding late.** Head-based sampling costs almost nothing but cannot see how a trace ends; tail-based sampling can keep the interesting traces but must hold everything in flight until it decides. - **Pull versus push.** Pulling turns a missed collection into a health signal but needs every target reachable; pushing works from short-lived or unreachable senders and leaves the backend unable to tell a dead sender from a quiet one. - **Index at write versus scan at read.** Indexing every word makes open-ended search fast and every write expensive; indexing only stream labels makes writes cheap and punishes queries that do not narrow by stream and time.
Some shapes recur under different names across the sections; recognising them helps you place an unfamiliar question quickly. - **Bound the value set.** Allowlists with a catch-all bucket, normalised paths and values grouped into ranges all replace an open-ended dimension with a fixed vocabulary. When a dimension cannot be bounded, it belongs on a per-request signal instead of a metric. - **Carry identity forward.** Trace context, logging context, sampling verdicts and correlation IDs are all request identity travelling with the work. Every boundary the work crosses, whether network, thread or queue, needs an explicit hand-over. - **Merge raw, compute once.** Bucket counts, sums and totals combine across instances; quantiles and ratios are derived after merging, never averaged from per-instance results. - **Keep a baseline beside the filter.** Whenever a policy keeps only the unusual (errors, slow traces, high-severity lines), a small unbiased sample kept alongside it is what lets you say what normal looks like. - **Count what you drop.** Sampling rules, drop filters and overflowing buffers should report how much they discarded, or a gap in the data is indistinguishable from a quiet system.
explore
- Telemetry Signals11 questions
- Metrics, Logs, and Traces Trade-offs4 questions
- Events and Continuous Profiling4 questions
- Choosing the Right Signal3 questions
- Metrics Concepts19 questions
- Counter, Gauge, Histogram, Summary4 questions
- Labels and Cardinality4 questions
- Histogram Buckets and Percentile Pitfalls4 questions
- Push vs Pull Collection4 questions
- RED and USE Checklists3 questions
- Logging Concepts19 questions
- Structured Logging4 questions
- Log Levels in Production3 questions
- Aggregation Pipeline Stages4 questions
- Index-Based vs Label-Based Storage4 questions
- Retention and Cost Control4 questions
- Tracing and Sampling16 questions
- Spans, Traces, and Context Propagation4 questions
- Correlation IDs and Exemplars4 questions
- Head vs Tail Sampling4 questions
- Agent vs Library Instrumentation4 questions
- Backend Developerrole anchor
- Data Engineerrole anchor
- DevOps / SRE Engineerrole anchor
- DevSecOps Engineerrole anchor
- Forward Deployed Engineerrole anchor
- Full Stack Developerrole anchor
- JS Full Stack Developerrole anchor
- Java Backend Developerrole anchor
- Java SDETrole anchor
- Kotlin Backend Developerrole anchor
- Kubernetestool anchor
- MLOps Engineerrole anchor
- Network Engineerrole anchor
- QA Engineerrole anchor
- Server-Side Game Developerrole anchor
- Software Architectrole anchor
questions
65 · 4 sectionsWhich of metrics, logs and traces do you check first for a rising error rate, one customer's failed request, or one endpoint's doubled p99?
basics
~20 sMetrics first for a rising error rate: they show whether it moved, when, and how far. Per-request records first for one named customer's request. A trace first for a single endpoint's latency, to see which downstream hop grew.
What do metrics, logs and traces each capture, and what does each one's data shape make it unable to answer?
basics
~20 sMetrics are numbers aggregated over an interval, so they show trends but never an individual request. Logs are discrete timestamped records of single events. Traces are causally linked trees showing one request's path across services.
In a flame graph from a sampling profiler, what do a frame's width and the vertical stacking each mean?
basics
~20 sWidth is the share of collected stack samples containing that frame — its share of the sampled resource, not elapsed time and not a call count. Vertical stacking is caller-to-callee ancestry, merged across every sample.
What does a sampling profiler record, and how does a profile differ from a trace or a metric?
basics
~20 sA sampling profiler interrupts the program at a fixed rate and records the call stack at that instant, merging identical stacks into counts. The result attributes a resource — CPU time or allocated bytes — to code paths rather than to requests.
How does monitoring differ from observability, and what must a system already be emitting for the second to work?
basics
~20 sMonitoring answers questions you predicted, from views and conditions built in advance for known failure modes. Observability is the property that lets you answer questions nobody anticipated, by slicing retained per-request detail along dimensions you did not choose beforehand.
What identifies one time series in a dimensional metrics system, and how does adding a metric label change the total count?
basics
~20 sA time series is the metric name plus its complete label set; change any label value and it is a different series. Adding a label multiplies the series count by that label's distinct values rather than adding to it.
What makes a metric a counter rather than a gauge, and why is a counter read as a rate over a window?
basics
~20 sA counter only ever increases and returns to zero when its publishing process restarts; a gauge is a level that moves either way. Because a counter's absolute value depends on process uptime, you read its rate over a window.
Why can't you average the p99 latencies of ten replicas to get the fleet's p99?
basics
~20 sA percentile is an order statistic over a set of observations, not a quantity that adds. Averaging per-replica p99s ignores how much traffic each served and describes no real request. Merge the underlying bucket counts first, then estimate the quantile once.
What does a pull-based metrics collector get for free, and what does it demand of every target?
basics
~20 sA pull-based collector already holds the list of what should exist, so every failed poll is itself a liveness signal, and one setting fixes the whole fleet's sampling resolution. In exchange each target must stay routable and answerable at any instant.
Under the RED checklist, how do you instrument a request-serving service, and what counts as an error?
basics
~20 sRED means emitting three things for a request-serving component: request rate, failed requests, and a duration distribution. Rate and errors come from one counter carrying an outcome dimension, and "error" must be defined in writing before the count means anything.
What does each conventional log level from TRACE to FATAL mean, and what test separates WARN from ERROR?
basics
~20 sLog levels are a contract about who must act. ERROR means something failed and a human should look; WARN means something recovered but degraded; INFO records state changes; DEBUG and TRACE are developer detail, off by default in production.
What does a log backend that full-text indexes every line at ingest buy you, and what does it cost?
basics
~20 sA full-text index built at ingest makes any word in any line searchable without declaring it first, and makes aggregations fast. You pay twice: CPU on the write path for every record, and index bytes stored alongside the logs.
What does structured logging give a log consumer that a formatted message string cannot?
basics
~20 sStructured logging emits each line as named, typed fields instead of one formatted sentence. Ingest no longer has to guess where each value starts and ends, and queries can filter, compare and aggregate on a field rather than searching raw text.
In a log aggregation pipeline, what does each stage — collect, parse, enrich, route, store — do, and which decisions can no later stage undo?
basics
~20 sCollect reads bytes out of the producing process, parse turns text into named fields, enrich attaches context the emitter lacked, route chooses destinations, store indexes and serves. Collect losses are unrecoverable: an unread line exists in no other copy.
What drives the cost of a centralized log platform, and which of those does shortening retention actually reduce?
basics
~20 sFour things drive a log bill: bytes ingested, the parsing and index-building done on them at write time, stored bytes multiplied by replicas and retained days, and query load. Shortening retention shrinks only the third.
What does a trace ID on every log line buy you, and what makes it go missing?
basics
~20 sA trace ID turns log search into an exact lookup: one filter returns every line every service wrote for one request. It goes missing when no span is active, when context is lost on an async hop, or when the logger never reads it.
In distributed tracing, what identifiers does a span carry, and why is a span an interval rather than a point in time?
basics
~20 sA span carries a trace id shared by every span of one request, its own span id, and its parent's span id. It stores a start time and a duration because it measures an operation that lasts, not an instant.
In head-based trace sampling, where is the keep-or-drop decision made, and why is it nearly free?
basics
~20 sHead-based sampling decides at the start of a trace, before any span is recorded. The first service rolls the dice once and stamps the verdict onto the outgoing request, so a dropped trace costs nothing downstream.
What can an automatic instrumentation agent see inside a running service, and what is it structurally blind to?
basics
~20 sAn automatic instrumentation agent hooks call sites it has a plugin for - web handlers, database drivers, message clients - so it produces spans at library boundaries only. Domain logic between those boundaries stays one unattributed gap in the trace.
In distributed tracing, what must travel with an outbound request for the callee to continue the same trace, and what breaks when one hop drops it?
basics
~20 sThe request must carry the trace id, the id of the span making the call, and whether the trace is being recorded. A hop that drops them makes the callee start a brand-new trace, taking every service below it with it.