skip to content

Two instances of a job read the Consul KV key `config/app/limits`, each modify it, and each write it back with `PUT /v1/kv/config/app/limits`. How does a check-and-set write using the entry's ModifyIndex stop one from silently clobbering the other, and how does the API report a CAS that did not apply?

level: middleimportance: should knowfreq 42%

answer

  1. optimistic, not locked
  2. you assert what you last saw
  3. the entry already carries a version token
  4. rejection is not an HTTP error
  5. re-read before you retry

basics

~20 s

Pass the ModifyIndex you read as ?cas=<index> on the PUT. Consul applies the write only if the key's current ModifyIndex still matches; otherwise it rejects it — returning HTTP 200 with a body of false, not an error status.

solid answer

~50 s

Consul gives you optimistic concurrency rather than locking. Read the key, keep its `ModifyIndex`, and write with `PUT /v1/kv/config/app/limits?cas=1904`. The write applies only if the entry's current `ModifyIndex` is still 1904; if the other instance got there first the index has moved and Consul refuses. The catch that trips people up is how refusal is reported: the response is **HTTP 200 with the body `false`**, not a 409 or a 412. A client that checks only the status code believes every write succeeded and you are back to last-write-wins. So parse the body, and on `false` re-read, re-apply your change to the fresh value and retry a bounded number of times. `?cas=0` is the special create-only case: the write applies only if the key does not exist. On the CLI it is `consul kv put -cas -modify-index=1904 config/app/limits '...'`, which exits non-zero when the CAS fails.

code

bash · 23 lines
bash
#!/usr/bin/env bash
# Read-modify-write with check-and-set and bounded retries.
set -euo pipefail
ADDR=http://127.0.0.1:8500
KEY=config/app/limits

for attempt in 1 2 3 4 5; do
  entry=$(curl -s "${ADDR}/v1/kv/${KEY}")
  idx=$(printf '%s' "$entry" | jq -r '.[0].ModifyIndex')
  cur=$(printf '%s' "$entry" | jq -r '.[0].Value' | base64 -d)
  next=$(printf '%s' "$cur" | jq -c '.max_conns += 10')

  # 200 OK is returned for both outcomes: the BODY is true or false.
  ok=$(curl -s -X PUT --data "$next" "${ADDR}/v1/kv/${KEY}?cas=${idx}")
  if [ "$ok" = "true" ]; then
    echo "applied on attempt ${attempt}"
    exit 0
  fi
  sleep "0.$((RANDOM % 5 + 1))"
done

echo "gave up: key too contended" >&2
exit 1

go deeper

for a junior

Know that a plain KV write replaces the whole value, so two concurrent writers can lose an update, and that Consul offers a conditional write keyed on the entry's ModifyIndex.

for a middle

Explain the read-modify-CAS-retry loop and the reporting trap: a rejected write comes back as HTTP 200 with the body false. Say why the ModifyIndex works as a version token at all.

for a senior

Show the operational judgment around it — bounded jittered retries, re-reading on each attempt, and the atomicity boundary: one key per CAS, the transaction endpoint when a change spans several. Distinguish CAS from mutual exclusion.

for a principal

Own the key layout that decides contention: one large document means every editor fights over one index, while a key per setting removes contention but gives up cross-setting atomicity. Decide which writers may be unconditional at all.

## Why an unconditional write is dangerous `PUT /v1/kv/<key>` replaces the whole value. There is no merge, no field-level update and no append. So the read-modify-write cycle two processes perform concurrently is a classic lost update: both read index 1904, both compute a new document from it, both write, and the second write silently erases the first one's change. Nothing errors. Nothing logs. The only evidence is a setting that mysteriously reverted. ## Check-and-set Every entry carries `CreateIndex` (the Raft index at which it was created) and `ModifyIndex` (the index of its last modification). Because Raft indexes only ever increase, `ModifyIndex` is a perfect version token. Supplying it on the write turns the operation into a conditional one: ``` GET /v1/kv/config/app/limits -> ModifyIndex 1904, Value ... PUT /v1/kv/config/app/limits?cas=1904 ``` The leader applies the write only if the entry's current `ModifyIndex` is exactly 1904. If a competing write landed in between, the index is now 1905 and yours is rejected. This is optimistic concurrency control: no locks are taken, no one waits, and contention is paid for only when it actually happens, as a retry. `?cas=0` has a dedicated meaning — apply only if the key does not exist. That is the primitive for "initialise this key exactly once", and it is race-free in a way that "read, see 404, write" is not. ## The reporting trap The body of a KV write is the literal JSON `true` or `false`, and both come back with HTTP 200. A failed CAS is a *business* outcome, not a transport failure, so nothing in the status line reflects it. Any client that follows the usual "raise on non-2xx, otherwise carry on" pattern will treat a rejected write as a success. This is the most common defect in hand-rolled Consul KV integrations, and it is invisible until two writers overlap — which is exactly the situation CAS was added for. Always parse the body. ## The retry loop The correct shape is read, modify, CAS, and on `false` go round again with a freshly read value: ``` for attempt in 1..N: entry = get(key) next = mutate(entry.value) if put(key, next, cas=entry.ModifyIndex): return OK backoff() fail("contended") ``` Two details make this production-grade. Re-read on every attempt — retrying with the stale index just fails again forever. And bound the attempts with a small jittered backoff: unbounded retry under heavy contention is a livelock that burns CPU on both sides and produces no completed writes. ## When one key is not enough CAS is per key. If a change must span several keys — a version marker plus the payload it describes — two independent CAS writes leave a window where a reader sees one applied and the other not. The transaction endpoint `PUT /v1/txn` takes a batch of KV operations, including conditional ones, and applies them atomically: either every operation lands or none does. It is also how you express "write these three keys only if this fourth key still has the index I expect". CAS is also not a mutual-exclusion primitive. It stops a lost update on one key; it does not stop two workers running the same job at the same time. That needs a session-backed lock, a different mechanism. ## Deletes and the same discipline `DELETE /v1/kv/<key>?cas=<index>` exists for the same reason: deleting a key you last saw at index 1904 should not silently discard an update that landed at 1905. Automation that removes configuration is exactly where you want the conditional form, since an unconditional recursive delete is the operation people most regret. ## Choosing the granularity How much you pack into one value decides how much contention you get. A single `config/app/limits` document containing twelve settings means twelve independent editors contending on one index. Splitting it into a key per setting removes almost all contention and makes each write naturally atomic — at the cost of losing all-or-nothing updates across settings, which is when you reach for the transaction endpoint instead.

  • What does ?cas=0 mean, and why is it not the same as checking for a 404 first?
    It means "apply only if this key does not exist", evaluated atomically by the leader. Checking with a GET and then writing leaves a window in which another writer creates the key between your two calls, so both callers believe they initialised it and one silently overwrites the other. With cas=0 exactly one caller gets true, which is the standard way to seed a default or elect an initialiser exactly once.
  • You need to update three related keys together. How?
    Use the transaction endpoint, PUT /v1/txn, which applies a batch of KV operations atomically — all commit or none do — and supports conditional operations inside the batch. Three separate CAS writes leave observable intermediate states where a reader sees one key updated and the others not, which is precisely the inconsistency a config consumer will act on.
  • Does CAS stop two workers from running the same job simultaneously?
    No. CAS protects one key from a lost update; it says nothing about who is allowed to do work. Mutual exclusion needs a session-backed lock, where a key is acquired with a session Consul invalidates if the holder dies. Using CAS as a lock gives you a value nobody clobbered and two workers happily running in parallel.
  • Your retry loop spins forever under load. What is wrong?
    Almost always one of two things: retrying with the original index instead of re-reading, so the CAS can never succeed; or unbounded retries with no backoff against a hot key, which livelocks every writer. Bound the attempts, add jittered backoff, and if a key really is that contended, split the document so writers stop competing for one index.

It is the same contract as a non-fast-forward git push: you state which commit you believed the branch was on, and the server rejects the push if someone else moved it since. You fetch, replay your change onto the new tip and try again.

saying these in an interview costs you the question

  • Checks only the HTTP status and misses the false body
  • Expects a failed CAS to return 409 or 412
  • Retries the CAS with the stale index instead of re-reading
  • Calls CAS a lock and relies on it for mutual exclusion
  • Assumes a PUT merges fields rather than replacing the value

context

resolving links…