Why can controller-runtime's mgr.GetClient() return stale or unexpectedly expensive reads, and when should a controller use mgr.GetAPIReader() instead?

middleshould knowfreq 46%

answer

reads and writes take different paths
read-your-own-write gap
first Get starts an informer
list and watch RBAC, cluster-wide memory
DisableFor, ByObject, live reader

basics

~20 s

mgr.GetClient() serves Get and List from informer caches but sends writes to the API server, so a read can lag your own write, and the first read of a new kind starts a cluster-wide informer. mgr.GetAPIReader() reads live and caches nothing.

solid answer

~50 s

The manager's default client splits reads from writes: Create, Update, Patch and Delete go straight to the API server, while Get and List for typed objects come from a shared informer cache. That has three consequences. First, a Get right after your own Create can return NotFound until the watch event arrives. Second, by default the first read of a kind the controller never watched starts an informer for that kind across every namespace, so the controller needs list and watch RBAC, and the cache holds every object of that kind in memory. Missing RBAC makes the read block until its context ends. Third, unstructured objects are read live unless configured otherwise. Use mgr.GetAPIReader() for rare reads that must be current or that you do not want cached, such as a one-off Secret lookup. For hot paths, prefer scoping the cache with cache.Options or listing kinds in client.CacheOptions.DisableFor.

The manager's default client splits reads from writes: `Create`, `Update`, `Patch` and `Delete` go straight to the API server, while `Get` and `List` for typed objects come from a shared informer cache. That has three consequences. First, a `Get` right after your own `Create` can return NotFound until the watch event arrives. Second, by default the first read of a kind the controller never watched starts an informer for that kind across every namespace, so the controller needs `list` and `watch` RBAC, and the cache holds every object of that kind in memory. Missing RBAC makes the read block until its context ends. Third, unstructured objects are read live unless configured otherwise. Use `mgr.GetAPIReader()` for rare reads that must be current or that you do not want cached, such as a one-off Secret lookup. For hot paths, prefer scoping the cache with `cache.Options` or listing kinds in `client.CacheOptions.DisableFor`.

go deeper

for a junior

Remember that the manager's client reads from memory and writes to the API server, and that a second reader, GetAPIReader, always goes live.

for a middle

Explain the read-after-write gap and the lazy informer: which RBAC verbs it needs, why it is cluster-wide by default, and how DisableFor or ByObject changes that.

for a senior

Diagnose an operator whose memory or permissions grew unexpectedly, trace it to a newly cached kind, and pick between scoping, DisableFor and live reads based on how often and how broadly the kind is read.

for a principal

Set platform conventions for operators on shared clusters: required cache scoping, a ban on cluster-wide Secret caches, and a budget for live reads against API server fairness limits.

Two readers behind one manager

A controller-runtime manager exposes two ways to read objects:

mgr.GetClient() returns a client.Client. Its writes (Create, Update, Patch, Delete, and Status() writes) are HTTP calls to kube-apiserver. Its reads (Get, List) are answered from the manager's shared informer cache, an in-memory copy of objects kept current by a watch.
mgr.GetAPIReader() returns a client.Reader, which has only Get and List. Every call is a live request to the API server. Nothing is cached, and it cannot write.

How the cache itself is fed (list, watch, resource versions) belongs to the watch-mechanics topic. This topic is about how controller-runtime wires the cache into your client, and what that wiring costs.

Consequence 1: your own writes are not visible yet

A Create returns once the API server has persisted the object, but the cache changes only when the watch event reaches it. A GameSession controller that creates a Deployment and then immediately runs r.Get on it can see NotFound. Treating that as "missing, create again" produces an AlreadyExists error. Workable patterns:

Use the object returned by the write call, which already carries the server's response.
Return and let the Deployment's own event (routed through Owns) trigger the next pass.
Use mgr.GetAPIReader() for the rare check that must be current, for example confirming that a port allocation record exists before handing it to a match server.

Consequence 2: lazy, cluster-wide informers

The cache is created on demand. By default, the first Get or List of a kind that no controller watches starts a new informer for that kind, and the read blocks until that informer has synced. Without any scoping, the informer lists and watches the kind across all namespaces. On a 210-node multi-tenant platform cluster that has real costs:

Memory: one r.Get of a single Secret makes the operator hold every Secret in the cluster, for example 46,803 objects, including values it will never read.
RBAC: the ServiceAccount needs list and watch on that resource, not just get. Without them the informer cannot sync, the read waits until its context ends and then fails with a timeout, and the reflector keeps logging forbidden errors.
Surprise: the code change was one line in Reconcile, but the operator's memory and permission footprint changed across the whole cluster.

Setting cache.Options{ReaderFailOnMissingInformer: true} makes such a read return an error instead of silently starting an informer, which turns the surprise into a visible failure.

Scoping and bypassing the cache

Tool	Where it is set	Effect
`cache.Options.DefaultNamespaces`	manager options	Caches only the listed namespaces
`cache.Options.ByObject`	manager options	Per-kind `Label`, `Field`, `Namespaces` or `Transform`
`client.CacheOptions.DisableFor`	`ctrl.Options.Client.Cache`	Listed kinds are always read live
`client.CacheOptions.Unstructured`	same	`false` (the default) reads unstructured objects live
`builder.OnlyMetadata`	on `Owns`/`Watches`	Caches only `metadata`, read via `PartialObjectMetadata`
`mgr.GetAPIReader()`	call site	One-off live read, never cached

A useful rule of thumb:

Kinds you watch (anything in For, Owns or Watches) are already cached, so read them from the client.
Kinds you read rarely and selectively, such as a single referenced Secret, are better served by DisableFor or the API reader.
Kinds you read often and broadly should be cached, but scoped with a label selector.

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
	Scheme: scheme,
	Cache: cache.Options{
		ByObject: map[client.Object]cache.ByObject{
			&corev1.ConfigMap{}: {Label: labels.SelectorFromSet(labels.Set{"games.example.com/profile": "true"})},
		},
	},
	Client: client.Options{
		Cache: &client.CacheOptions{DisableFor: []client.Object{&corev1.Secret{}}},
	},
})

The cost of reading live

The API reader is not free either. Each call is a request against the API server's rate limits and priority-and-fairness budget. A reconciler that calls GetAPIReader().List on every pass over thousands of GameSession objects adds load that the cache exists to avoid. Keep live reads for:

correctness checks where a stale answer would do real damage,
kinds the controller must not cache for memory or secrecy reasons,
startup or migration code that runs once.

Summary

The split client makes controllers cheap: hundreds of reads per second hit memory, not the API server. The price is eventual consistency on reads and a cache footprint that grows with every kind you touch. Know which kinds your manager caches, and scope them deliberately.

## Two readers behind one manager A controller-runtime **manager** exposes two ways to read objects: - `mgr.GetClient()` returns a `client.Client`. Its **writes** (`Create`, `Update`, `Patch`, `Delete`, and `Status()` writes) are HTTP calls to kube-apiserver. Its **reads** (`Get`, `List`) are answered from the manager's shared **informer cache**, an in-memory copy of objects kept current by a watch. - `mgr.GetAPIReader()` returns a `client.Reader`, which has only `Get` and `List`. Every call is a live request to the API server. Nothing is cached, and it cannot write. How the cache itself is fed (list, watch, resource versions) belongs to the watch-mechanics topic. This topic is about how controller-runtime wires the cache into your client, and what that wiring costs. ## Consequence 1: your own writes are not visible yet A `Create` returns once the API server has persisted the object, but the cache changes only when the watch event reaches it. A `GameSession` controller that creates a Deployment and then immediately runs `r.Get` on it can see **NotFound**. Treating that as "missing, create again" produces an `AlreadyExists` error. Workable patterns: 1. Use the object returned by the write call, which already carries the server's response. 2. Return and let the Deployment's own event (routed through `Owns`) trigger the next pass. 3. Use `mgr.GetAPIReader()` for the rare check that must be current, for example confirming that a port allocation record exists before handing it to a match server. ## Consequence 2: lazy, cluster-wide informers The cache is created **on demand**. By default, the first `Get` or `List` of a kind that no controller watches starts a new informer for that kind, and the read **blocks until that informer has synced**. Without any scoping, the informer lists and watches the kind across all namespaces. On a 210-node multi-tenant platform cluster that has real costs: - **Memory**: one `r.Get` of a single Secret makes the operator hold every Secret in the cluster, for example 46,803 objects, including values it will never read. - **RBAC**: the ServiceAccount needs `list` and `watch` on that resource, not just `get`. Without them the informer cannot sync, the read waits until its context ends and then fails with a timeout, and the reflector keeps logging forbidden errors. - **Surprise**: the code change was one line in `Reconcile`, but the operator's memory and permission footprint changed across the whole cluster. Setting `cache.Options{ReaderFailOnMissingInformer: true}` makes such a read return an error instead of silently starting an informer, which turns the surprise into a visible failure. ## Scoping and bypassing the cache | Tool | Where it is set | Effect | |---|---|---| | `cache.Options.DefaultNamespaces` | manager options | Caches only the listed namespaces | | `cache.Options.ByObject` | manager options | Per-kind `Label`, `Field`, `Namespaces` or `Transform` | | `client.CacheOptions.DisableFor` | `ctrl.Options.Client.Cache` | Listed kinds are always read live | | `client.CacheOptions.Unstructured` | same | `false` (the default) reads unstructured objects live | | `builder.OnlyMetadata` | on `Owns`/`Watches` | Caches only `metadata`, read via `PartialObjectMetadata` | | `mgr.GetAPIReader()` | call site | One-off live read, never cached | A useful rule of thumb: - Kinds you **watch** (anything in `For`, `Owns` or `Watches`) are already cached, so read them from the client. - Kinds you read **rarely and selectively**, such as a single referenced Secret, are better served by `DisableFor` or the API reader. - Kinds you read **often and broadly** should be cached, but scoped with a label selector. ```go mgr, err := ctrl.NewManager(cfg, ctrl.Options{ Scheme: scheme, Cache: cache.Options{ ByObject: map[client.Object]cache.ByObject{ &corev1.ConfigMap{}: {Label: labels.SelectorFromSet(labels.Set{"games.example.com/profile": "true"})}, }, }, Client: client.Options{ Cache: &client.CacheOptions{DisableFor: []client.Object{&corev1.Secret{}}}, }, }) ``` ## The cost of reading live The API reader is not free either. Each call is a request against the API server's rate limits and priority-and-fairness budget. A reconciler that calls `GetAPIReader().List` on every pass over thousands of `GameSession` objects adds load that the cache exists to avoid. Keep live reads for: - correctness checks where a stale answer would do real damage, - kinds the controller must not cache for memory or secrecy reasons, - startup or migration code that runs once. ## Summary The split client makes controllers cheap: hundreds of reads per second hit memory, not the API server. The price is **eventual consistency on reads** and a **cache footprint** that grows with every kind you touch. Know which kinds your manager caches, and scope them deliberately.

Your operator's memory jumped from about 90 MiB to over 1 GiB after one release that added a single Secret lookup. What happened, and how do you fix it?
The cached client lazily started a cluster-wide Secret informer, so the operator now holds every Secret in the cluster. Either list `corev1.Secret` in `client.CacheOptions.DisableFor` so lookups go live, read it through `mgr.GetAPIReader()`, or restrict the cache with `cache.Options.ByObject` using a label or namespace scope. Also trim the RBAC back to what the narrower access needs.
Why does controller-runtime read unstructured objects live by default?
`client.CacheOptions.Unstructured` defaults to false, so `unstructured.Unstructured` reads bypass the cache. Unstructured access is typically used for arbitrary or dynamically discovered kinds, and silently starting informers for each of them would multiply memory use and RBAC needs. Set the option to true only when you really do watch those kinds.

saying these in an interview costs you the question

A Get right after Create through mgr.GetClient() always returns the new object
The cached client reads only the kinds listed in For and Owns and errors otherwise
get permission alone is enough for the cached client to read a new kind
mgr.GetAPIReader() can also create and update objects
Reading through the API reader on every reconcile is free because it is just a GET

Why can controller-runtime's mgr.GetClient() return stale or unexpectedly expensive reads, and when should a controller use mgr.GetAPIReader() instead?

§ answer

basics

solid answer

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Two readers behind one manager

Consequence 1: your own writes are not visible yet

Consequence 2: lazy, cluster-wide informers

Scoping and bypassing the cache

The cost of reading live

Summary

▾common follow-ups2 questions

▾red flags5 flags

Two readers behind one manager

Consequence 1: your own writes are not visible yet

Consequence 2: lazy, cluster-wide informers

Scoping and bypassing the cache

The cost of reading live

Summary

answer

go deeper