A case repository refreshes its stored tracker links only when the tracker calls it back on a change. Why does it still drift out of agreement over time, and what does a periodic reconciliation sweep add?

A tracker-side callback is a performance optimization. It tells you about a change in seconds instead of at the next sweep. It is not, and cannot be, the thing that makes two stores agree, because a consumer has no way to notice a message it never received. ## What a callback stream cannot tell you Three distinct gaps, and they fail differently: - **The delivery you never got.** A restart, an expired credential, a lapsed certificate, a receiver that returned errors until the sender gave up. The change happened, the announcement did not arrive, and the stored value is now wrong while looking exactly as trustworthy as one updated a second ago. Staleness has no visible signature. - **The change that announces nothing you subscribed to.** Visibility changes are the classic case: nobody edited the item, but the integration account lost access to its project, so every link into it now resolves to a not-found. Bulk administrative operations and project archival often behave the same way. - **Order.** Announcements are not guaranteed to arrive in the order the changes happened. Applying an older state after a newer one leaves the repository stably wrong until the next change comes along. None of these produce an error on the receiving side, so a callback-only design fails silently and indefinitely. The first person to notice is usually a human reading a link whose state has not matched the ticket for a fortnight. ## Two sweeps, doing two different jobs A **reconciliation sweep** re-derives the repository's copy from the tracker instead of waiting to be told. It comes in two shapes, and using one where you needed the other is the common mistake: 1. **Watermark sweep (incremental).** Ask the tracker for everything changed since the timestamp of the last successful sweep, and apply it. Cost is proportional to churn, so it can run often. It catches every missed update — including ones you never knew you missed — and catches nothing about items that were deleted or that left your visibility, because those do not appear in a list of changes. 2. **Full sweep (resolve every stored link).** Walk your own links and resolve each identifier against the tracker. Cost is proportional to how many links you store rather than to churn, so it runs on a long cadence. This is the only pass that can discover **absence**: the link whose target no longer answers. The watermark itself is the subtle part. Store the tracker's notion of modification time, not your clock, and advance it only after the sweep's writes have committed — a watermark advanced before the work is durable turns one crash into a permanent hole. Overlapping the window slightly on each run is cheap insurance against boundary rounding, and is harmless as long as applying the same change twice is a no-op. ## Both paths must end in the same code The callback handler and the sweep must converge on identical state from identical input. The reliable way to guarantee that is structural: neither applies anything itself. Both produce "the tracker says this item now looks like this", and one shared apply function decides what changes locally. That makes every apply **idempotent** — running it twice changes nothing the second time — which is what lets the sweep overlap the callback stream freely, resume after a partial failure, and be re-run on demand after an incident without anyone having to reason about what might get applied twice. ## A sweep has a cost, and needs a budget A full sweep is the most expensive thing the integration does, and it lands on a system shared with everyone else in the company. The mechanics that make it acceptable are ordinary and non-negotiable: - Read in **batches** rather than one item per request, so the cost scales with batch count instead of link count. - **Chunk and pace** the work, back off exponentially when the tracker signals throttling, and treat throttling as an instruction to slow down rather than an error to retry immediately. - Make it **resumable**, so a sweep interrupted midway continues instead of starting over. - Run it **off-peak**, and treat its cadence as a configurable operational choice, because the right interval depends on how much drift the organization can tolerate. ## The sweep is also the only drift meter you have Instrument how many links each sweep actually changed. In a healthy integration that number sits near zero: the callbacks got there first, and the sweep merely confirms what you already believed. A sweep that keeps finding real changes is telling you the fast path is broken — an endpoint unreachable, a subscription silently dropped, an account whose access was narrowed. That signal exists nowhere else, which is why the sweep earns its cost even when it is usually a no-op.