Grafana offers a variable type called 'Ad hoc filters'. Explain how it reaches a panel's query differently from a normal query variable, what it needs from the data source to work, and the situations where it produces surprising results.
answer
- never referenced as $var — injected implicitly
- bound to ONE data source; other panels unfiltered
- needs plugin support: injection + key/value suggestion
- totals/baselines get filtered too → ratios lie
- default filters ≠ security
basics
~20 sAn ad-hoc filter variable is never referenced with $name. It holds a list of key/operator/value filters that Grafana injects automatically into every query sent to its chosen data source on that dashboard. It needs the data source plugin to support both the injection and the key/value suggestion API.
solid answer
~1 minA normal variable only affects a query if the query mentions it. An **ad-hoc filters** variable is the opposite: you bind it to one data source, and Grafana appends its key/value predicates to *every* query that dashboard sends to that data source — no `$var` reference anywhere. The UI gives the user a key drop-down and a value drop-down, populated by the data source's own tag-keys/tag-values (or equivalent) API, so it is an exploratory 'slice by anything' control rather than a curated one. Requirements and caveats: - The **data source plugin must implement it**; coverage is uneven and the operator set (`=`, `!=`, regex-match, and in newer versions `IN`/`NOT IN`) varies. - It is **scoped to one data source**. On a mixed dashboard, panels backed by other sources silently ignore the filter — the user thinks the whole dashboard is filtered when half of it is not. - It **applies to queries that should not be filtered** — a panel showing a fleet-wide total gets the filter too, so the total quietly stops being a total. - A key the user picks may not exist on every metric/stream in the query, and depending on the backend that turns a panel empty rather than unfiltered. - Key/value lookups can be expensive on high-cardinality backends. Use it for exploration dashboards; use explicit variables where the filtering semantics must be exact.
go deeper
Know that it is a filter bar bound to a data source and that you do not reference it by name in queries.
Explain the implicit injection by the plugin, the key/value suggestion API, and the single-data-source scope.
Lead with the two dangerous cases — mixed-source dashboards and unfilterable baseline panels — plus lookup cost on high-cardinality keys and the ad-hoc-versus-explicit choice.
Decide policy: exploration dashboards may use it, contractual/SLO dashboards may not; defaults are convenience only, isolation is enforced at the data path.
## The mechanism Ad hoc filters are the one variable type with **implicit** application. You create the variable, choose a data source, and Grafana renders a filter bar where the user adds `key operator value` clauses. Those clauses are then merged into the query *by the data source plugin* as it builds the outgoing request — added as label matchers on a metrics query, as filter clauses on a log or search query, as `WHERE` predicates on sources whose plugin supports it. The panel's own query text is untouched and contains no reference to the variable. This is why the type exists: it turns a fixed dashboard into an exploratory one. A user who notices a spike can slice by `region`, then by `version`, without editing a single panel — and the selection is captured in the URL like any other variable, so the sliced view is shareable. ## What the data source must provide Two capabilities, and a plugin can have neither, one, or both in different quality: 1. **Injection** — the plugin must know how to fold arbitrary key/value predicates into its query language. 2. **Suggestion** — the key and value drop-downs come from the data source's metadata API (tag keys/tag values, label names/label values, field mappings). Without it, the user must type keys blind. Coverage is genuinely uneven and has changed across Grafana releases: label-based metrics stores, log stores and search backends are the strongest; SQL sources historically had no support and gained it unevenly. Newer Grafana versions also let you configure a fixed key source (so keys come from a chosen query or a static list rather than an expensive lookup) and support richer operators including set membership. Do not promise a behaviour without checking the plugin on the version in use. ## Where it surprises people **Mixed-data-source dashboards.** The variable is bound to exactly one data source. Panels backed by anything else receive no filter at all. Visually nothing indicates this, so a user filters to `region=eu` and reads a dashboard where half the panels are still global. This is the single most damaging failure mode because it produces confidently wrong conclusions rather than obvious breakage. **Panels that must not be filtered.** Any panel meant to show a denominator, a fleet-wide total, or a comparison baseline gets the same predicate injected. Ratios silently become 1.0; 'this service versus everything' becomes 'this service versus this service'. If a dashboard has such panels, either move them to a different data source, or do not use ad-hoc filters on that dashboard. **Keys that do not exist everywhere.** A filter on a key that only some series carry will, on most backends, exclude the series lacking that key entirely rather than leaving them alone. The user sees empty panels and assumes an outage. **Cost.** Value suggestion on a high-cardinality key is one of the most expensive metadata queries a backend can be asked, and it runs interactively as the user types. On a large installation, opening the value drop-down for something like a request-id-shaped label can be slow enough to look hung. **Interaction with other filtering.** If the panel query already constrains the same key, you now have two predicates on the same dimension. Depending on the operator that is either redundant (`=` twice on the same value), contradictory (empty result), or narrowing in a way the dashboard author never intended. **Provisioning.** In dashboards-as-code, the ad-hoc variable can carry default filters, so a provisioned dashboard can arrive pre-scoped. That is useful for multi-tenant dashboards — but it is a *default*, not a security control: the user can remove it, and anything that must not be visible has to be enforced at the data source or by row-level permissions, never by a dashboard filter. ## Ad hoc versus explicit variables Choose ad hoc when the dashboard is for exploration, the dimensions are not known in advance, and every panel legitimately shares the same scope. Choose explicit query variables when the filtering must be precise and per-panel — when some panels are baselines, when different panels filter on different keys, or when the data source does not support ad hoc well. A common hybrid is explicit variables for the two or three dimensions the dashboard is *about*, plus an ad-hoc variable for opportunistic slicing on the rest. ## Interview framing Lead with the structural difference — implicit injection into every query for one data source, no `$var` reference — because that is the fact everything else follows from. Then the data-source dependency, then the two dangerous cases (mixed sources, unfilterable panels). Finish with 'and it is not an authorisation mechanism'.
- Why is an ad-hoc filter with a default value not a tenant-isolation mechanism?It is a dashboard-side default that the user can edit or delete from the filter bar, and it only affects the one data source it is bound to. Real isolation has to be enforced where the data is served — per-tenant credentials, query-time label enforcement at the proxy, or row-level rules in the store. Treat the filter as convenience, never as a boundary.
- A user adds an ad-hoc filter and half the dashboard goes empty. How do you diagnose it?Check whether the filtered key exists on the series those panels query — most backends drop series lacking the key rather than ignoring the predicate. Then confirm the panels actually use the bound data source; panels on other sources are unaffected, which produces the mirror-image confusion. The panel inspector's query tab shows whether the predicate was injected at all.
saying these in an interview costs you the question
- Trying to reference an ad-hoc variable as `$filters` inside a query.
- Assuming it filters every panel on the dashboard regardless of data source.
- Using default ad-hoc filters as a tenancy or permissions control.
- Forgetting that baseline/total panels are filtered too, so ratios silently become meaningless.
- Claiming every data source supports it — support and operator sets vary by plugin and version.