A reconcile loop clones observed state with maps.Clone, mutates the clone, and never converges — how do you find the cause?

## The shape of the bug A reconcile loop reads the observed state of a resource, computes the desired state, compares the two, and writes only if they differ. When it never converges, the loop is either writing on every tick (the diff never comes out empty) or never writing at all (the diff always comes out empty even though the world is wrong). Both symptoms fall out of the same root cause more often than anything else: **the "copy" of the observed state was never independent of the observed state.** ## Why the clone was shallow Both stdlib clone helpers copy exactly one level: - `maps.Clone(m)` allocates a new map and copies the key/value pairs with ordinary assignment. If the value type is `[]string`, each *header* is copied and each backing array is shared. If the value type is another map or a pointer, the pointee is shared. - `slices.Clone(s)` allocates a new array of `len(s)` elements and copies them with ordinary assignment. For `[]Resource` where `Resource` has a `Tags []string` field, every struct is copied but every `Tags` array is shared. So `desired := maps.Clone(observed)` followed by a mutation of `desired["web"]` writes straight through into `observed["web"]`. The diff is then computed between two structures that just moved together, and it comes out empty — the loop concludes there is nothing to do while the cluster stays wrong. ## Why in-place helpers make it worse `slices.Delete`, `slices.Compact`, `slices.Replace` and `slices.Insert` (when capacity allows) all work **inside the backing array**. They do not merely fail to protect the original; they actively rewrite it, and since Go 1.22 they also zero the slots they vacate. Concretely: ```go observed := map[string][]string{"web": {"prod", "canary"}} copied := maps.Clone(observed) copied["web"] = slices.Delete(copied["web"], 1, 2) ``` `copied["web"]` is now `[prod]`, and `observed["web"]` is `[prod ""]` — length 2, second element zeroed. The snapshot you were going to diff against has been corrupted in a way that looks like data loss upstream, which is what sends people hunting in the wrong system at 3am. ## The nil-versus-empty half of the problem The other way a reconcile loop fails to settle is a diff that is *never* empty. `reflect.DeepEqual` distinguishes a nil slice or map from an empty non-nil one: they are deeply equal only if both are nil or both are non-nil. A desired state built in code as `map[string]string{}` and an observed state decoded from an empty payload as `nil` will compare unequal forever, so the loop writes the same object on every tick — a hot loop with no visible change, which shows up as write pressure or rate limiting rather than as a bug. `maps.Equal` and `slices.Equal` do not have that rule: they compare pairs and elements, and a nil container simply has none, so nil and empty compare equal. Whichever you pick, pick it because it matches the contract — if "absent" and "present but empty" mean different things to your API, the comparison must preserve that; if they do not, use the comparison that ignores it. ## How to diagnose it 1. **Write the aliasing test.** Build the observed state, clone it, mutate every nested container in the clone, then assert with `reflect.DeepEqual` that the original still equals a freshly-built expected value. A shallow field fails this instantly and names itself in the diff. Do the reverse too — mutate the original, assert the clone is unchanged — because a single shared field only shows up in one direction if your mutation happens to be one-sided. 2. **Log the diff, not the decision.** Print the two structures the comparison actually received, not a boolean. A diff that is empty when the world is wrong points at aliasing; a diff that is permanently non-empty on a field that looks identical points at nil versus empty. 3. **Check the comparison function.** Grep for `reflect.DeepEqual` in the reconcile path. It is both the slowest option (reflection on every tick) and the one with the nil/empty rule. 4. **Then decide the fix, not just the patch.** Two defensible fixes exist: deep-clone the nested containers explicitly at the boundary where the snapshot enters the loop, or treat the observed snapshot as immutable and never mutate it at all, building the desired state from scratch. The second is usually better — a defensive clone is a cost paid on every tick and one forgotten nested field brings the bug back, whereas an immutability convention can be reviewed. ## What to write down afterwards The rule worth putting in the package doc is short: **`Clone` in the standard library is one level deep, and the `slices` mutators are in place.** Anything that must survive being handed to code that mutates it needs either a real deep copy written for that type, or a documented contract that the caller must not write to it.