Watch and Informer Mechanics

## The mechanism Kubernetes never takes locks on objects. Instead every object has a `metadata.resourceVersion` that changes on every write. When you send a full update, you send the object you read, resourceVersion included. The API server compares it with the stored version: - match: the write is applied and the version advances; - mismatch: **409 Conflict**, nothing is written. This is optimistic concurrency control, and it converts a lost-update bug into a visible error. Without it, a read-modify-write pair from two controllers would silently clobber one another: both read version 5, both write their own full object, and the second erases the first's field. ## Why it fires so often in controllers Controllers usually read from an informer cache, which is eventually consistent. If the object changed a moment ago, your cached copy already holds an outdated resourceVersion, so the very first update attempt conflicts. Popular objects also attract many writers — status updates, finalizer additions, label edits from several operators. Conflicts on hot objects are normal, not exceptional; the design assumption is that you retry. ## Handling it well **Retry with a fresh read.** The essential detail is that each retry must re-fetch the object from the API server (a live GET, not the informer's Lister) and reapply the mutation to that fresh copy, because only that gives you a current resourceVersion. client-go's `retry.RetryOnConflict(retry.DefaultRetry, fn)` implements the loop; you supply a function that gets, mutates and updates. Bound the retries — infinite loops on a permanently contended object are a real outage mode — and let the controller's workqueue requeue with backoff if retries are exhausted. **Prefer a patch.** A full update carries the entire object and therefore conflicts on any concurrent change anywhere in it. A patch carries only your intent: - *strategic merge patch* — the Kubernetes-aware default for built-in types, merging lists by key rather than replacing them; - *JSON merge patch* — simple object merge, replaces lists wholesale; - *JSON patch* — an operation list, and the only one that can express a test-then-set precondition; - *server-side apply* — declarative, with per-field ownership; it conflicts only when you contest a field another field manager owns. Patches usually need no resourceVersion at all, so they are the pragmatic choice for narrow mutations like adding a label, a finalizer or an annotation. ## Know when you still want the conflict If your change depends on the value you read — incrementing a counter, appending to a list, or making a decision from the current status — a blind patch can corrupt data. There you *want* optimistic concurrency: read, compute, update with resourceVersion, and let 409 force a recomputation on the new value. ## Anti-patterns - **Clearing resourceVersion to force the write.** It turns the update into an unconditional overwrite and reintroduces lost updates. Reviewers should treat it as a defect. - **Retrying without re-reading.** Guaranteed to fail repeatedly and to hammer the API server. - **Retrying the wrong errors.** 409 Conflict means retry; 422 Invalid, 403 Forbidden and 404 Not Found will not improve with repetition. - **Updating spec and status in one call for resources with a status subresource.** They are separate endpoints; mixing them either drops changes or produces avoidable conflicts. - **Unbounded retry loops on a contended object**, which starve the workqueue. ## The mindset Treat 409 as normal control flow in a system where many autonomous loops write concurrently. The question a strong candidate answers is not "how do I stop it happening" but "is my change independent enough to patch, or dependent enough that I must re-read and recompute".