You are looking at a span inside Grafana's trace view and want to jump to that request's log lines. How is that jump configured on the Tempo data source, and why does it so often come back empty?

seniorshould knowfreq 30%

answer

tracesToLogsV2 on the Tempo datasource
tags map span attributes → log labels (service.name → service_name)
spanStartTimeShift / spanEndTimeShift widen the window
filterByTraceID needs the id to be filterable, not just in the body
Read the query Grafana built — the bug is usually visible there

basics

~20 s

On the Tempo data source's trace-to-logs section: pick the logs data source, map span/resource attributes to log labels (tags), set a time-window shift around the span, and optionally filter by trace or span id. It comes back empty mostly from attribute-to-label name mismatches or too narrow a time window.

solid answer

~60 s

Trace-to-logs is configured on the Tempo (trace) data source, in the trace-to-logs section — tracesToLogsV2 in provisioning YAML. You set: the target logs data source UID; a tags mapping that converts span or resource attributes into log-store labels, because the naming conventions differ (service.name on a span versus service_name or job on a log stream); spanStartTimeShift / spanEndTimeShift to widen the query window around the span (for example -5m / 5m); and toggles for filterByTraceID / filterBySpanID. For anything non-trivial you supply a custom query using ${__span.traceId}, ${__span.spanId} and ${__tags} interpolation.

Empty results almost always come from one of four causes: the tag mapping produces a label that does not exist in the log store; the log store has the trace id only inside the line, so a label filter can never match and you need a line filter; the time window is too tight because log ingest timestamps drift from span timestamps; or the logs were dropped or the service simply never logged for that request. The same configuration family covers trace-to-metrics (and, in newer versions, trace-to-profiles), using the same tag interpolation.

Trace-to-logs is configured on the **Tempo (trace) data source**, in the trace-to-logs section — `tracesToLogsV2` in provisioning YAML. You set: the **target logs data source UID**; a **tags** mapping that converts span or resource attributes into log-store labels, because the naming conventions differ (`service.name` on a span versus `service_name` or `job` on a log stream); **spanStartTimeShift / spanEndTimeShift** to widen the query window around the span (for example `-5m` / `5m`); and toggles for **filterByTraceID / filterBySpanID**. For anything non-trivial you supply a **custom query** using `${__span.traceId}`, `${__span.spanId}` and `${__tags}` interpolation. Empty results almost always come from one of four causes: the tag mapping produces a label that does not exist in the log store; the log store has the trace id only inside the line, so a label filter can never match and you need a line filter; the time window is too tight because log ingest timestamps drift from span timestamps; or the logs were dropped or the service simply never logged for that request. The same configuration family covers **trace-to-metrics** (and, in newer versions, trace-to-profiles), using the same tag interpolation.

code

text · 11 lines

text

tempo datasource
  jsonData.tracesToLogsV2:
    datasourceUid: loki-main
    tags:
      - key: service.name      value: service_name
      - key: k8s.namespace.name  value: namespace
    spanStartTimeShift: "-5m"
    spanEndTimeShift:   "5m"
    filterByTraceID: true
    # escape hatch when the id is in the body, not a label:
    # customQuery: <line filter on ${__span.traceId}>

go deeper

for a junior

Know the feature exists on the trace data source and that it needs a mapping between span attributes and log labels.

for a middle

Name the four knobs — target UID, tags mapping, time shift, id filtering — and explain why the names differ between the two signals.

for a senior

Debug it: read the generated query first, then work through label mismatch, id not filterable, window too tight, nothing logged, retention gap.

for a principal

Argue for standardising service/environment identity across pipelines so the mapping is trivial, and for provisioning the configuration as code with pinned UIDs.

Direction matters

Grafana treats the two directions as separate features owned by different data sources. Log → trace is a derived field on the log data source. Trace → logs is a setting on the trace data source. Candidates who conflate them usually cannot debug either.

The configuration surface

On the Tempo data source (provisioned as jsonData.tracesToLogsV2):

datasourceUid — the logs data source to query. Must be stable across environments, so pin it in the datasource provisioning file.
tags — a list of mappings, each { key, value }, where key is the span or resource attribute and value is the log label to use. This exists because the two worlds name things differently: tracing conventions use dotted attributes such as service.name, k8s.pod.name, deployment.environment, while log stores carry labels such as service_name, pod, namespace, job. Without a mapping, Grafana would build a query filtering on a label nobody has.
spanStartTimeShift / spanEndTimeShift — durations added to the span's start and end to form the query range. Defaults are tight; real deployments widen to minutes.
filterByTraceID / filterBySpanID — when the log store can filter on the id, restrict to exactly that request or that span.
customQuery / query — free-form query text with interpolation: ${__span.traceId}, ${__span.spanId}, ${__span.tags}, ${__tags}, and ${__trace.traceId}. This is the escape hatch when the automatic construction cannot express what you need — for instance filtering on the line rather than on a label.

The sibling settings tracesToMetrics and (in newer versions) tracesToProfiles work identically: same tag mapping, same interpolation, different target. The point is the same — jump from one individual request to the aggregate around it or to the other signal about it.

Why it comes back empty

Diagnose in this order, cheapest first:

Look at the query Grafana actually built. The jump opens the target pane with a concrete query. Read it. Nine times out of ten the answer is visible there: a label that does not exist, or a filter on the wrong value.
Label mismatch. The tag mapping named a label the log store does not have, or has under a different name in that particular pipeline. Different collector configurations map resource attributes to labels differently, so a mapping that works for one service can fail for another in the same cluster.
Trace id not filterable. filterByTraceID builds a label-style filter. If the log pipeline keeps the trace id only inside the log body (or as structured metadata rather than a label), that filter matches nothing. Fix with a custom query that does a line filter on ${__span.traceId} instead.
Time window. Log timestamps come from the logging call or, worse, from ingest time; span times come from the tracer. Clock skew, batching in the log agent and late arrival all push log lines outside a tight window. Widening the shift to a few minutes is the standard fix, at the cost of a broader query.
Nothing was logged. A fast successful request often produces spans and no log lines at all. An empty result can be correct.
Retention asymmetry. If traces are retained longer than logs, older traces jump into a window the log store has already dropped.

Design consequences

The recurring theme is that the join key and the join dimensions must be agreed across pipelines: the same service identity, the same environment identity, the same trace id encoding, present in both stores, and filterable in both. That agreement is a platform decision made at collector/agent configuration time, not something Grafana can paper over. Grafana's tag mapping is a translation layer for a mismatch you would rather not have; when you own both pipelines, it is cheaper to standardise the label names and keep the mapping trivial.

Finally, prefer provisioning this configuration as code. Trace-to-logs settings clicked into a single instance are invisible to review, differ between environments, and break the moment a data source UID differs.

## Direction matters Grafana treats the two directions as separate features owned by different data sources. Log → trace is a **derived field on the log data source**. Trace → logs is a setting on the **trace data source**. Candidates who conflate them usually cannot debug either. ## The configuration surface On the Tempo data source (provisioned as `jsonData.tracesToLogsV2`): - **datasourceUid** — the logs data source to query. Must be stable across environments, so pin it in the datasource provisioning file. - **tags** — a list of mappings, each `{ key, value }`, where `key` is the span or resource attribute and `value` is the log label to use. This exists because the two worlds name things differently: tracing conventions use dotted attributes such as `service.name`, `k8s.pod.name`, `deployment.environment`, while log stores carry labels such as `service_name`, `pod`, `namespace`, `job`. Without a mapping, Grafana would build a query filtering on a label nobody has. - **spanStartTimeShift / spanEndTimeShift** — durations added to the span's start and end to form the query range. Defaults are tight; real deployments widen to minutes. - **filterByTraceID / filterBySpanID** — when the log store can filter on the id, restrict to exactly that request or that span. - **customQuery / query** — free-form query text with interpolation: `${__span.traceId}`, `${__span.spanId}`, `${__span.tags}`, `${__tags}`, and `${__trace.traceId}`. This is the escape hatch when the automatic construction cannot express what you need — for instance filtering on the line rather than on a label. The sibling settings `tracesToMetrics` and (in newer versions) `tracesToProfiles` work identically: same tag mapping, same interpolation, different target. The point is the same — jump from one individual request to the aggregate around it or to the other signal about it. ## Why it comes back empty Diagnose in this order, cheapest first: 1. **Look at the query Grafana actually built.** The jump opens the target pane with a concrete query. Read it. Nine times out of ten the answer is visible there: a label that does not exist, or a filter on the wrong value. 2. **Label mismatch.** The tag mapping named a label the log store does not have, or has under a different name in that particular pipeline. Different collector configurations map resource attributes to labels differently, so a mapping that works for one service can fail for another in the same cluster. 3. **Trace id not filterable.** `filterByTraceID` builds a label-style filter. If the log pipeline keeps the trace id only inside the log body (or as structured metadata rather than a label), that filter matches nothing. Fix with a custom query that does a line filter on `${__span.traceId}` instead. 4. **Time window.** Log timestamps come from the logging call or, worse, from ingest time; span times come from the tracer. Clock skew, batching in the log agent and late arrival all push log lines outside a tight window. Widening the shift to a few minutes is the standard fix, at the cost of a broader query. 5. **Nothing was logged.** A fast successful request often produces spans and no log lines at all. An empty result can be correct. 6. **Retention asymmetry.** If traces are retained longer than logs, older traces jump into a window the log store has already dropped. ## Design consequences The recurring theme is that the join key and the join dimensions must be *agreed across pipelines*: the same service identity, the same environment identity, the same trace id encoding, present in both stores, and filterable in both. That agreement is a platform decision made at collector/agent configuration time, not something Grafana can paper over. Grafana's tag mapping is a translation layer for a mismatch you would rather not have; when you own both pipelines, it is cheaper to standardise the label names and keep the mapping trivial. Finally, prefer provisioning this configuration as code. Trace-to-logs settings clicked into a single instance are invisible to review, differ between environments, and break the moment a data source UID differs.

Why does trace-to-logs need a time shift at all, when the span already has a start and end time?
Span times come from the tracer's clock at the moment of execution; log timestamps come from the logging call, or in bad pipelines from ingest time after agent batching. Clock skew between hosts, batching delay and late arrival all push relevant lines outside the exact span interval. A window of a few minutes on each side trades a broader, slightly more expensive query for the property that the lines you want are actually inside it.
The tag mapping works for one service and returns nothing for another in the same cluster. What is the likely cause?
The two services' telemetry pipelines map resource attributes to log labels differently — different collector or agent configuration, a different log format, or one service setting the resource attribute and the other not. Grafana builds the same query for both, so the one whose labels do not match that mapping returns nothing. The durable fix is to standardise the label set at the pipeline level rather than to add per-service mappings in Grafana.

saying these in an interview costs you the question

Configuring trace-to-logs on the log data source instead of the trace data source
Assuming span attribute names and log label names match, so no tag mapping is needed
Enabling filter-by-trace-id when the trace id only appears inside the log body
Treating an empty result as broken configuration when the service simply logged nothing for a fast request
Clicking the settings into one instance instead of provisioning them, so environments silently diverge

You are looking at a span inside Grafana's trace view and want to jump to that request's log lines. How is that jump configured on the Tempo data source, and why does it so often come back empty?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

Direction matters

The configuration surface

Why it comes back empty

Design consequences

▾common follow-ups2 questions

▾red flags5 flags

§ context

▾related questions6 questions

▾related topics3 topics

Direction matters

The configuration surface

Why it comes back empty

Design consequences

answer

go deeper

context