Describe how a Kubernetes client keeps an up-to-date view of a set of objects using the list-then-watch pattern, and what the resourceVersion value returned by the API server means in that flow.
answer
- LIST for snapshot, WATCH from its resourceVersion
- events: ADDED / MODIFIED / DELETED / BOOKMARK
- resourceVersion is an opaque resume cursor
- 410 Gone 'too old resource version' means relist
- resourceVersion=0 on list = cheap, possibly stale
basics
~20 sThe client LISTs the objects once, notes the collection's resourceVersion, then opens a WATCH starting after that version and receives ADDED/MODIFIED/DELETED events as a stream. resourceVersion is an opaque cursor into the API server's change history, not a number to compare or interpret.
solid answer
~60 sA client first issues a **LIST** to get a full snapshot; the response's `metadata.resourceVersion` marks the exact point in the change stream that snapshot represents. It then opens a **WATCH** with `resourceVersion=<that value>`, and the API server streams every change after it as typed events — ADDED, MODIFIED, DELETED (plus BOOKMARK) — each carrying the affected object and its own resourceVersion. The client applies events to its in-memory copy, so after the initial list it never re-lists in steady state. resourceVersion is **opaque**. It is backed by an etcd revision today, but the contract is: pass it back unmodified and use it only to resume from a point. Do not compare two objects' values numerically, do not do arithmetic on it, and do not persist it across clusters. The watch connection can break — a proxy timeout, an API server restart, or the requested version having aged out of the server's watch cache, which returns **HTTP 410 Gone** with `too old resource version`. The client's job is to handle that by re-listing to get a fresh snapshot and version, then watching again. That relist-and-rewatch loop is what client-go's Reflector does for you.
code
bash · 5 lineskubectl get --raw '/api/v1/namespaces/default/pods?limit=500' | jq '.metadata.resourceVersion'
kubectl get --raw '/api/v1/namespaces/default/pods?watch=true&resourceVersion=182773&allowWatchBookmarks=true'
# {"type":"ADDED","object":{..."resourceVersion":"182801"...}}
# on resume with an aged-out version:
# {"kind":"Status","code":410,"reason":"Expired","message":"too old resource version: 100 (182773)"}go deeper
Say that clients list once and then receive a stream of change events, and that resourceVersion marks where the stream resumes.
Walk the whole flow including event types, resume-after-disconnect, and the 410 relist path; state that resourceVersion is opaque.
Add operational nuance: the bounded watch cache window, consistent versus cached list reads, and why hand-rolled watch clients get 410 handling wrong.
Discuss the cost model — one LIST plus a stream per informer versus per-client polling — and the guarantees this gives controllers reasoning about staleness.
## The problem A controller needs a current view of, say, every Pod in a namespace. Polling with a LIST every second is correct but brutally expensive: full serialization of every object, repeated, from every client in the cluster. The list-then-watch pattern gives the same correctness at a tiny fraction of the cost. ## The flow 1. **LIST.** `GET /api/v1/namespaces/x/pods` returns a PodList. Its `metadata.resourceVersion` is the version of the *collection*: the point in the change stream at which this snapshot is consistent. 2. **WATCH from that point.** `GET /api/v1/namespaces/x/pods?watch=true&resourceVersion=<rv>` opens a long-lived streaming response (chunked HTTP, or an HTTP/2 stream). The server sends nothing for the past and then one JSON event per change: `{"type":"ADDED"|"MODIFIED"|"DELETED"|"BOOKMARK", "object": {...}}`. 3. **Apply events locally.** The client mutates its cached copy per event. Each delivered object carries its own resourceVersion; the client remembers the newest one it processed so it can resume from there. 4. **On disconnect, resume.** Reconnect with the last processed resourceVersion. If the server can still serve from that point, the stream continues with no gap and no relist. 5. **On 410 Gone, relist.** If the requested version is older than the server's retained history, the server returns `410 Gone: too old resource version`. The only correct response is to LIST again (fresh snapshot, fresh version) and rewatch. The client must resync its cache wholesale, because it cannot know what it missed. ## resourceVersion semantics The field appears on both individual objects and on list responses, and it is **opaque to clients**. The rules that matter: - Every write bumps the object's resourceVersion. Reads do not. - Values are only meaningful within one cluster; never store them long-term or compare across clusters. - Do not treat them as numbers. They are strings that happen to look numeric in the current implementation; ordering guarantees exist for the watch stream, not for arbitrary comparison of two objects. - Passing `resourceVersion=0` on a LIST means "any reasonably recent version you have, cheapest possible" — served from the API server's watch cache, possibly slightly stale. Omitting it requests a fully consistent read. That difference matters: `0` is cheap but can return data older than a write you just made. - The same value doubles as the optimistic-concurrency token on updates, which is why a stale one produces a 409 Conflict. ## Why 410 Gone exists The API server keeps a bounded window of recent changes in its watch cache (and etcd itself compacts old revisions). A client that was disconnected or paused for a long time may ask to resume from a version that has fallen out of that window. Rather than silently skipping events — which would leave the client permanently wrong — the server refuses with 410 and forces an honest resync. ## What the libraries do You rarely write this by hand. client-go's **Reflector** implements exactly this loop: list, record version, watch, apply events into a store, handle 410 by relisting, reconnect with backoff. Informers build on it. Writing it yourself is a known source of subtle bugs — mishandling 410, treating resourceVersion as an integer, or assuming a watch never breaks.
- What must a client do when a watch returns 410 Gone with 'too old resource version'?Discard the assumption that its cache is current, issue a fresh LIST to obtain a new snapshot and collection resourceVersion, rebuild the cache from that, and start a new watch from the new version. It must not simply retry the watch from the same old version, and it must not resume from an arbitrary newer version, because the events it missed are unknown.
- What is the difference between listing with resourceVersion=0 and listing without the parameter?resourceVersion=0 asks for any reasonably recent version and is typically served from the API server's watch cache, which is cheap but may be slightly behind. Omitting the parameter requests a fully consistent read at the current revision, which costs more. Using 0 right after a write can return data that does not yet include it.
Reading a shared document: download the current copy plus its revision number, then subscribe to the edit feed from that revision on. If the server has purged history that far back, you must download the whole document again.
saying these in an interview costs you the question
- Comparing resourceVersion values numerically or doing arithmetic on them
- Retrying a watch from the same version after 410 instead of relisting
- Assuming a watch connection is permanent and needs no reconnect logic
- Believing clients watch etcd directly rather than the API server
- Thinking resourceVersion=0 is always the safe default for reads