skip to content

You need to provision a Grafana data source that requires a password or API token, without committing the secret to the repository that holds the provisioning files. What mechanisms does Grafana give you, and what happens to that secret afterwards?

level: seniorimportance: should knowfreq 32%

answer

  1. secureJsonData (encrypted, write-only) vs jsonData (clear, readable)
  2. $VAR / $__env{} / $__file{/path} / $__vault{} interpolation
  3. GF_<SECTION>_<KEY> overrides any ini setting
  4. secret_key decrypts everything — back it up
  5. Datasources apply at startup or reload API, not the dashboard interval

basics

~20 s

Put the secret under secureJsonData in the datasource provisioning file, but reference it indirectly: Grafana interpolates environment variables and can read a value from a mounted file. Grafana encrypts secureJsonData in its database and never returns it through the API, so rotation means changing the source and reloading provisioning.

solid answer

~60 s

Secrets in a datasource provisioning file go under **`secureJsonData`** — never `jsonData`, which is stored and returned in clear. The value itself should be a reference, not a literal. Grafana interpolates provisioning files and its ini config with: plain `$VAR` / `${VAR}` environment variables, the explicit `$__env{VAR}` form, `$__file{/path/to/secret}` to read a mounted file (the natural fit for a Kubernetes Secret mounted as a volume), and `$__vault{…}` in editions that support it. Once loaded, Grafana **encrypts** `secureJsonData` in its own database using the instance's `secret_key`, and the values are **write-only**: the HTTP API and the UI show whether a secret is set, never its value. That has two consequences — you cannot recover a secret from Grafana, and your provisioning files remain the source of truth. Operationally: data source providers are applied **at startup or via the admin provisioning reload API**, not on the dashboard rescan interval, so rotating a mounted secret requires a reload or a restart. Also pin an explicit **`uid`** per data source so dashboards and internal links referencing it are portable, and remember that the whole ini config is overridable by `GF_<SECTION>_<KEY>` environment variables.

code

text · 12 lines
text
apiVersion: 1
datasources:
  - name: Mimir
    type: prometheus
    uid: prometheus-main          # pin: dashboards and links resolve by uid
    url: https://mimir.internal/prometheus
    editable: false
    jsonData:
      httpHeaderName1: "X-Scope-OrgID"
    secureJsonData:
      httpHeaderValue1: $__file{/etc/grafana/secrets/tenant-token}
      # or: ${TENANT_TOKEN} from the environment

go deeper

for a junior

Know that secrets go under secureJsonData and should come from an environment variable or a mounted file rather than being typed into the committed YAML.

for a middle

Name the interpolation forms including $__file{}, explain that secure values are encrypted and never returned, and that the header name/value split lives across the two buckets.

for a senior

Cover the lifecycle: startup-or-reload apply semantics, rotation procedure, deleteDatasources, editable, uid pinning and the empty-variable failure mode.

for a principal

Set the operating model — files plus secret store are the source of truth, the instance secret key is a recovery dependency, and rotation is an explicit, automated step rather than an assumption about mounts.

## Two buckets on a data source A provisioned data source separates its settings into `jsonData` (non-secret configuration: TLS options, HTTP method, scrape interval hints, exemplar links, derived fields) and **`secureJsonData`** (passwords, tokens, client secrets, private keys, custom HTTP header values). The difference is not cosmetic: `secureJsonData` is encrypted at rest in Grafana's database and is never returned by the API, while `jsonData` is stored plainly and comes back on every read. Putting a token in `jsonData` — for instance as part of a URL — leaks it to anyone who can read the data source through the API. Custom auth headers use a paired convention: the header *name* goes in `jsonData` (`httpHeaderName1`) and the header *value* in `secureJsonData` (`httpHeaderValue1`). ## Getting the secret in without committing it The file still has to say *something*. Grafana interpolates provisioning files (and `grafana.ini`) with several sources: - **Environment variables** — `$TOKEN`, `${TOKEN}`, or the unambiguous `$__env{TOKEN}`. Simple, works everywhere, but the value is visible in the process environment and in whatever orchestrator object set it. - **`$__file{/etc/secrets/token}`** — reads the file's contents at provisioning time. This is the best fit for Kubernetes: mount a Secret as a volume and reference the path. The secret never becomes an environment variable, and rotation is a file update. - **`$__vault{…}`** — direct lookup in a secret manager, in editions that support it. The same interpolation applies to the main config file, and additionally every ini setting can be overridden by an environment variable of the form `GF_<SECTION>_<KEY>` — for example the database password or the instance `secret_key`. That is how containerised Grafana is usually configured at all. ## What Grafana does with it afterwards At provisioning time Grafana writes the data source row and encrypts the `secureJsonData` fields with the instance-wide `secret_key` (or an envelope-encryption key hierarchy in newer versions, where a key-encryption key can come from a KMS). Three implications: 1. **Write-only.** No API call returns the plaintext. Reads return only whether each secure field is set. Good for exposure, inconvenient when someone hoped Grafana would serve as the record of what the token was — it must not. 2. **`secret_key` is load-bearing.** Restore the database into an instance with a different `secret_key` and every stored secret becomes undecryptable. It belongs in your backup and disaster-recovery plan, not only in the deployment manifest. 3. **Provisioning stays the source of truth.** Because you cannot read secrets back, the only sane operating model is that the files (plus the secret store they reference) define the state, and Grafana is a consumer. ## Lifecycle and rotation Dashboard providers rescan on an interval; **data source providers do not**. They are applied at startup, and otherwise when the admin provisioning reload endpoint is called. So rotating a token that is mounted as a file changes nothing until a reload or a restart happens. A rollout that relies on "the mounted secret updates automatically" will quietly keep using the old credential until the pod restarts. Decide explicitly: restart on rotation, or call the reload API from your rotation job. Also relevant: - **`deleteDatasources`** — a list of `{name, orgId}` entries removed before the rest of the file is applied; the way to retire a data source through provisioning rather than by hand. - **`editable`** — whether the data source can be modified in the UI. Default is locked, mirroring the dashboard read-only rule and for the same reason: UI edits are not written back to the file and would be lost at the next apply. - **`uid`** — pin it. Dashboards, derived fields, exemplar links, trace-to-logs targets and correlations all resolve data sources by UID, so a per-instance generated UID breaks every one of those in the next environment. ## Failure modes worth naming - A referenced environment variable is unset: the value interpolates to empty and the data source is provisioned with a blank credential, failing at query time with an auth error rather than at load time. - The secret contains characters that clash with YAML quoting, producing a subtly wrong value. - Someone edits the credential in the UI on an `editable: true` data source; it works until the next apply reverts it. - The provisioning file is committed with a literal secret "just for staging" and stays in git history forever. ## The short answer to give *Secrets go in `secureJsonData`, referenced via `$__file{}` from a mounted secret or an environment variable; Grafana encrypts them with its `secret_key` and never gives them back; data sources apply at startup or on the provisioning reload API, so rotation needs a reload; and pin the `uid` so everything that links to the data source stays portable.*

  • You rotate a token in a mounted Kubernetes Secret. Does the provisioned data source start using the new value?
    Not by itself. The file on disk updates once the kubelet propagates the change, but data source provisioning is applied at Grafana startup and otherwise only when the admin provisioning reload endpoint is invoked — unlike dashboards, there is no periodic rescan. So either the rotation job calls the reload API, or the rollout restarts the pods. Assuming the mount alone is enough leaves Grafana authenticating with the old credential until something restarts it.
  • Why can you not read a provisioned data source's password back out of Grafana, and why does that matter for disaster recovery?
    secureJsonData is encrypted at rest and treated as write-only: the API and UI report only whether a field is set. That limits exposure, but it means Grafana is not a record of your secrets — the provisioning files plus the referenced secret store are. It also makes the instance's secret key load-bearing: restoring the Grafana database into an instance configured with a different key leaves every stored secret undecryptable, so the key must be part of the backup and recovery plan.

saying these in an interview costs you the question

  • Putting a token in jsonData or in the URL, where it is returned by the API in clear
  • Committing a literal secret "just for staging" into the provisioning file
  • Expecting a rotated mounted secret to take effect without a provisioning reload or restart
  • Not pinning the data source uid, breaking every dashboard, derived field and exemplar link in other environments
  • Treating Grafana as a place to look up a secret later, when secure fields are write-only

context

resolving links…