Consul KV, Watches & Templated Config

## What the process is actually doing consul-template parses your template and extracts its *dependencies* — each `{{ service "web" }}`, `{{ key "config/app/mode" }}`, `{{ ls }}` or `{{ tree }}` call becomes a watched query. It opens a blocking query per dependency and keeps them open. When any query returns a new index, it re-renders the entire template in memory against the current view, compares the bytes with the existing destination file, and: - if identical, does nothing — no write, no command, which is why a wake-up on an unrelated change is harmless; - if different, writes the new content (to a temporary file in the destination directory, then renames it into place so a reader never sees a half-written config) and executes the template's `command`, subject to `command_timeout`. ```hcl template { source = "/etc/consul-template/nginx-upstreams.ctmpl" destination = "/etc/nginx/conf.d/upstreams.conf" command = "/usr/local/bin/reload-nginx" perms = 0644 backup = true error_on_missing_key = true wait { min = "2s" max = "10s" } } ``` ## Diagnosing the empty render The instinct is "Consul went down and the template rendered nothing". That is not how it behaves. If the agent is unreachable, the queries fail, consul-template retries with backoff and the destination file is left exactly as it was — stale, but serving. An empty list on disk therefore means the query *succeeded* and legitimately returned zero entries. The realistic causes: - **Every instance became unhealthy.** `{{ service "web" }}` returns only passing instances by default, so a bad check definition, a dependency outage or a deploy that failed its readiness gate empties the set. This is health checking doing its job and templating faithfully reporting it. - **The ACL token lost access.** A token whose policy no longer grants read on that service or KV prefix yields empty results rather than an obvious error in your rendered output. - **A filter that matches nothing.** A tag filter such as `service "canary.web"` renders empty the moment the tag disappears. The general lesson: **the template is a pure function of what Consul says, and "nothing" is a legal input.** Safety has to be written into the template and the reload path, not assumed. ## Making the render safe First, refuse to render nothing. Capture the result, check its length, and do not emit an empty block: ``` {{- $backends := service "web|passing" -}} {{- if gt (len $backends) 0 -}} upstream web { {{ range $backends }}server {{ .Address }}:{{ .Port }}; {{ end }} } {{- else -}} {{ scratch.Set "abort" true }} {{- end -}} ``` The cleanest version of this makes the *whole render* fail, so consul-template keeps the previous file rather than writing a valid-but-empty one. `error_on_missing_key = true` gives you the same property for KV lookups: an absent key aborts the render instead of interpolating an empty string into the middle of a directive. Second, never let an unvalidated file become live config. Point `command` at a small wrapper rather than directly at the reload: ```bash #!/usr/bin/env bash set -euo pipefail nginx -t # config test; non-zero exit aborts here nginx -s reload ``` Because a failing `command` is visible in consul-template's logs, a bad render becomes an alert instead of an outage. Combined with `backup = true` you also keep the previous file on disk for a manual fallback. ## Damping the churn `wait { min = "2s" max = "10s" }` is a quiescence timer: after the first change, consul-template waits for `min` of calm before rendering, and renders regardless after `max`. Without it, a rolling deploy that flaps twenty instances through the catalog triggers twenty renders and twenty reloads in a few seconds — each reload spawning a new worker generation while the old ones drain, which is how a routine deploy turns into a memory and connection spike. ## Testing it before it is live `consul-template -once` renders a single pass and exits, which is what you run in CI against a fixture Consul; `-dry` prints the rendered result to stdout instead of writing it, so you can diff a template change against real data without touching a file. A template change deserves the same review as a code change, because it *is* the code that produces your production routing table. ## Where the boundary sits Templating is one of three ways to get discovery data into a proxy, and it is worth knowing why you picked it. Rendering plus reload gives you the proxy's full configuration language at the cost of a reload on every change. Resolving service names through Consul's DNS interface avoids reloads but gives up per-instance metadata and inherits resolver caching behaviour. A runtime API — a socket that mutates server state in place without rewriting the config file — avoids both, where the proxy offers one. Templating is the general answer that works with any proxy, and its price is exactly the failure mode above: the file is authoritative, and a bad file is instantly authoritative too. envconsul is the same machinery aimed at process environment rather than files: it populates environment variables from KV and starts a child process, restarting or signalling it when values change — useful for applications that read config only at startup and cannot be taught to watch anything.