Observation API & Distributed Tracing

@Observed marks a method so Micrometer creates an Observation around each call (producing metrics and/or a trace span). For it to work you must register an ObservedAspect bean, which uses Spring AOP to wrap the annotated method.

The Observation API lets you wrap a piece of code once so Spring can produce monitoring signals from it. You create an Observation on an ObservationRegistry with a name and call observe(...) around your code.

Context propagation passes the current trace's IDs to the next service so their spans join one trace. With Micrometer Tracing, Spring auto-adds a W3C traceparent header carrying the trace ID, the caller's span ID, and a sampled flag.

It sets the fraction of traces that get recorded and exported, from 0.0 (none) to 1.0 (all). The default in Spring Boot is 0.1, meaning about 10% of traces are sampled.

An ObservationHandler is a listener that reacts to an Observation's lifecycle. onStart runs when the observation starts, onStop when it stops (e.g. to record timing), and supportsContext(context) decides whether this handler should handle a given observation at all.

You instrument code once with an Observation. The registry's handlers each react to its lifecycle events — one handler records a metric, another creates a trace span, another logs — so one instrumentation feeds all three pillars.

Baggage is arbitrary key/value data attached to the trace context and carried to downstream services alongside the trace IDs. Declare fields with management.tracing.baggage.remote-fields to propagate them, and correlation.fields to copy them into the logging MDC.

The sample-or-not decision is made once at the start (head) of a trace and stored on the context. It's carried to downstream services in the trace-context headers (e.g. the W3C traceparent flags byte), so every service in the trace makes the same keep/drop choice.

Create a span (tracer.nextSpan().name(...)), start it, open a scope with tracer.withSpan(span) so it becomes 'current', do work, then close the scope (try-with-resources) and call span.end(). Skipping end() loses timing/export; leaving a scope open leaks context onto later work on that thread.

KeyValues are key/value tags you attach to an Observation. Low-cardinality ones (few distinct values, like status) become metric tags and span tags. High-cardinality ones (many values, like user ID) go only on the span, not on metrics.

Pick one bridge. OTel bridge (micrometer-tracing-bridge-otel) is the modern default and exports via OTLP or Zipkin. Brave bridge (micrometer-tracing-bridge-brave) exports to Zipkin via zipkin-reporter-brave. Never add both.

Register an ObservationPredicate on the ObservationRegistry. It's a function of (name, context) returning true/false; if it returns false the observation becomes a no-op and nothing (no metric, no span) is recorded. Use it to filter out noise like /actuator paths.

Both encode the same trace/span IDs but in different headers. W3C uses one traceparent header (the Boot 3 default); B3 (from Zipkin/Brave) uses either a single b3 header or multiple X-B3-* headers. Set management.tracing.propagation.type to W3C or B3.

Spring instruments HTTP and other work as Observations. Tracing registers ObservationHandlers that turn each Observation into a span. For outbound calls a propagating handler injects trace headers; on the server side another handler extracts them, so the downstream span joins the same trace.

An ObservationConvention centralizes how an observation is named and tagged. It produces low-cardinality KeyValues (few distinct values — safe as metric tags/dimensions) and high-cardinality KeyValues (many distinct values like an id — attached to spans only, never to metric dimensions).

Observation.Context is a mutable data bag that travels through an Observation's lifecycle, carrying its name, KeyValues, error, and any custom fields. An ObservationConvention reads that Context to decide the Observation's name and KeyValues in one reusable place.

Context travels as message headers instead of HTTP headers. Spring's Kafka/RabbitMQ instrumentation injects traceparent/baggage into producer record headers and extracts them on the consumer. Where no auto-instrumentation exists, use the Propagator API to inject on send and extract on receive manually.

Sampling 100% is accurate but expensive: more CPU/memory per request, more network to the collector, and much higher storage/backend cost. Lower probabilities cut cost but risk missing the rare errored or slow request. You balance budget against how much visibility you need.

For simple cases set management.tracing.sampling.probability. For custom logic, define your own Sampler bean (Brave's brave.sampler.Sampler or OpenTelemetry's io.opentelemetry.sdk.trace.samplers.Sampler), which overrides the property. Brave offers RateLimitingSampler and per-path samplers you can compose.

At 0.1 only ~10% of traces are exported by design — that's expected, not a bug. The sampling decision is made once at the trace root and propagated downstream, so all services agree. Verify the endpoint, that ids still appear in logs (proving propagation works), and consider raising probability or using tail sampling in the collector.

Create an Observation with a convention and context, start it (onStart fires), open a scope so nested work sees trace context (onScopeOpened), run the work, record errors (onError), then stop it (onStop records timing/finishes the span). Predicates decide up front whether it runs at all; conventions decide naming/tags; handlers do the actual recording.

Create with createNotStarted, start() it, openScope() to bind it to the thread (so child spans/logs correlate), run work, call error(e) on failure, close the scope, then stop(). Forgetting the scope breaks correlation; forgetting stop() leaks. observe() does all this safely; async work needs context propagation.

Restrict baggage to a small, approved allow-list of small non-sensitive fields; forbid secrets/PII since baggage is forwarded everywhere and logged. Standardize field names and propagation format fleet-wide, strip baggage at public trust boundaries, and watch header-size/overhead on high-volume paths.

Head-based sampling decides before the outcome is known, so it randomly drops errors and slow requests and can't guarantee capturing rare problems. At scale you add tail-based sampling in a collector, rate limiting, per-route rates, force-sample-on-debug, and lean on metrics/exemplars so tracing isn't your only signal.