In Patroni, how do ttl, loop_wait and retry_timeout bound the leader key, and what does the primary do when renewal fails?
answer
- a key that expires unless renewed
- 30, 10 and 10 seconds
- a sum that must fit inside ttl
- demote before the lease lapses
basics
~20 sPatroni's primary holds a leader key in the DCS that expires after ttl (30 s) and renews it every loop_wait (10 s), retrying for retry_timeout (10 s). If renewal fails it demotes PostgreSQL, timed to finish before the key expires.
solid answer
~50 sThe leader key lives at `/<namespace>/<scope>/leader` (namespace `/service` by default) and is written with an expiry of `ttl` seconds; on etcd v3 it is bound to a lease. The primary's HA loop renews it every `loop_wait` seconds, and DCS or PostgreSQL calls are retried for up to `retry_timeout`, so a shorter blip demotes nobody. The defaults are 30, 10 and 10 seconds (minimums 20, 1 and 3), and they must satisfy `loop_wait + 2 * retry_timeout <= ttl`; Patroni logs a warning and lowers `loop_wait` if you break it. That rule leaves the primary time to fail a renewal and demote before its key lapses. A failed update makes it stop PostgreSQL in immediate mode and restart it as a replica; an unreachable DCS demotes it too, unless `failsafe_mode` lets it stay or the cluster is paused.
code
bash · 6 lines# Show the dynamic configuration stored in the DCS for the payment cluster
patronictl -c /etc/patroni/patroni.yml show-config paygw
# Raise ttl for a slower DCS while keeping loop_wait + 2 * retry_timeout <= ttl (10 + 30 = 40)
patronictl -c /etc/patroni/patroni.yml edit-config paygw \
-s ttl=40 -s loop_wait=10 -s retry_timeout=15 --forcego deeper
Remember that Patroni's primary owns a leader key in etcd or a similar store, and that the key expires unless the primary keeps renewing it.
Give the defaults 30, 10 and 10, the rule loop_wait plus twice retry_timeout within ttl, and the two demotion paths: failed update and unreachable DCS.
Explain why shrinking ttl alone causes needless demotions, how primary_start_timeout changes crash handling, and when pause or failsafe_mode keep a primary running.
Frame the timers as a trade between failover speed and false failovers on your DCS's real latency, and decide who owns tuning them across environments.
## What the leader key is Patroni turns a PostgreSQL primary and its streaming standbys into a self-managing cluster by keeping one fact in a **distributed configuration store (DCS)** — etcd, Consul, ZooKeeper, Exhibitor or the Kubernetes API (the built-in Raft backend is deprecated): which member is the leader. That fact is the **leader key**, stored at `/<namespace>/<scope>/leader`, where `namespace` defaults to `/service` and `scope` is the cluster name. The key holds the leader's member name and carries an expiry, so it disappears on its own if nobody renews it. On etcd v3 Patroni attaches it to a **lease** and creates it in a transaction that only succeeds when the key does not exist yet, so two members cannot both create it. A PostgreSQL node may run as primary only while its Patroni agent owns this key and can keep updating it. Everything below follows from that rule. ## The three timers These are **dynamic settings**: they live in the cluster's `/config` key in the DCS, apply to every member, and are changed with `patronictl edit-config` or `PATCH /config` on the REST API. Values under `bootstrap.dcs` in `patroni.yml` are written to `/config` when the cluster is first initialised, so editing them later does nothing to a running cluster. | Setting | Meaning | Default | Minimum | |---|---|---|---| | `ttl` | lifetime of the leader key, in seconds | 30 | 20 | | `loop_wait` | seconds the HA loop sleeps between iterations | 10 | 1 | | `retry_timeout` | how long DCS and PostgreSQL operations are retried | 10 | 3 | The documentation describes `ttl` as the time before the automatic failover process starts, and `retry_timeout` as the length of DCS or network trouble that will *not* demote the leader. ## The rule that binds them Patroni requires `loop_wait + 2 * retry_timeout <= ttl`; the defaults give 10 + 2 × 10 = 30. If an edit breaks the rule, Patroni logs a warning and lowers `loop_wait`; only when even a `loop_wait` of 1 cannot satisfy it does it set `loop_wait` to 1 and lower `retry_timeout` as well. The rule exists for this sequence: 1. The primary renews the key, which is now valid for `ttl` seconds. 2. It sleeps up to `loop_wait`, then tries to renew again. 3. A slow DCS makes that attempt retry for up to `retry_timeout`. 4. When the attempt finally fails, the primary still needs time to stop acting as primary before the key lapses and a replica can take it. Shrinking `ttl` alone speeds up failover but makes a leader that is briefly slow to reach the DCS demote itself for no good reason. ## What the primary does when renewal fails - **Update failed** (`failed to update leader lock` in the log): the primary stops PostgreSQL in **immediate** mode, without a checkpoint and without releasing the key, then starts it again as a replica. Writes stop at once. - **DCS unreachable altogether**: one node cannot tell 'etcd is down' from 'I am cut off', so Patroni assumes a **network partition**. It demotes PostgreSQL and keeps it read-only until the DCS answers again. - **`failsafe_mode: true`**: in the DCS-unreachable case only, the primary may stay primary if every member listed in the `/failsafe` key acknowledges it over the REST API. - **Paused cluster** (`patronictl pause`): Patroni leaves PostgreSQL running as primary even when the update fails, because maintenance mode switches off automatic actions. ## How long an automatic failover takes - **Host or network loss**: the key expires at most `ttl` seconds after the last successful renewal. The replicas then run a **leader race**, and the healthiest eligible member takes the key and promotes. - **PostgreSQL crashed, Patroni alive**: the agent still renews the key while it tries to restart PostgreSQL for `primary_start_timeout` (default 300 seconds). The documented worst case is `loop_wait + primary_start_timeout + loop_wait`; with `primary_start_timeout: 0` Patroni fails over as soon as it detects the crash, accepting that an asynchronous replica may miss recent commits. ## Tuning judgement - Keep the defaults until you have measured how long your DCS takes to answer under load. - Change `ttl`, `loop_wait` and `retry_timeout` together, and check the rule yourself rather than relying on Patroni's automatic adjustment, which only shows up as a warning in the log. - A watchdog, if used, expires `safety_margin` seconds before `ttl` (5 by default), so it moves when `ttl` moves. - The general theory of leases and why they prevent two primaries is a separate topic; the Patroni-specific part is these settings and what the agent does when the renewal fails.
- Why does Patroni demote the primary when the DCS is merely unreachable instead of letting it keep writing?From one node, a dead etcd and a partition that isolates this node look identical. In the partition case the other side can see the leader key expire, take it and promote a replica. Demoting before the key can lapse is the only way to guarantee a single writer without more information; `failsafe_mode` supplies that information by asking every known member directly over the REST API.
- What changes if PostgreSQL on the primary crashes but the Patroni agent keeps running?Patroni still renews the leader key, so no replica can take over. It tries to restart PostgreSQL in place for up to `primary_start_timeout` (300 seconds by default) before letting a failover happen. Setting it to 0 fails over immediately, trading a possible quick local recovery for a promotion that may lose asynchronously replicated commits.
The leader key works like a hotel key card coded to expire at checkout: the primary must re-code it at the front desk every loop_wait seconds, and if it cannot reach the desk it leaves the room before the card stops working, because the hotel may already be handing the room to someone else.
- ttl is how often the primary renews the leader key.
- The primary keeps accepting writes until a replica has promoted itself.
- Setting ttl to 5 seconds gives a five-second automatic failover.
- Editing ttl under bootstrap.dcs in patroni.yml changes it on a running cluster.
- Any DCS hiccup, however short, makes Patroni demote the primary.