A team discovers that a projection's event-handling logic has been silently miscalculating a customer's lifetime-spend column for six months. What are the mechanics of fixing this by rebuilding the projection from the event log, and what must the event handlers guarantee for that rebuild to produce correct results?

## Why a rebuild is possible at all The single property that makes projection rebuilds possible at all is that **the event log, not the projection, is the source of truth**: a projection is explicitly a derived, disposable artifact, so if its logic was wrong, the fix isn't to patch the corrupted data in place - it's to discard the projection entirely and regenerate it from the immutable facts that never changed. ## The mechanics, step by step Mechanically, a rebuild has a few concrete steps. 1. **First, the handler bug is fixed and deployed** - in the example, whatever logic accumulates lifetime spend (perhaps it double-counted refunded orders, or missed a currency conversion) is corrected in code. 2. **Second, the projection's storage is reset**: the target table (or index, or cache) is truncated or dropped and recreated, and the projector's checkpoint is reset to position zero. 3. **Third, a fresh catch-up subscription is started** against the full event log, replaying every event from the beginning through the now-corrected handler, exactly the same mechanism used to bootstrap a brand-new projection. Once replay reaches the tail, the projection is back to serving live traffic with corrected data, and typically only then is the corrected projection cut over to serve production reads, often via a **blue-green swap** so the buggy version keeps serving until the corrected one is verified. ## Why not a migration script This mechanism exists because it is dramatically safer and simpler than trying to patch six months of accumulated errors with a targeted data-migration script. A migration script has to reverse-engineer exactly which rows are wrong and by how much, is itself new code that can have its own bugs, and offers no guarantee it matches what the corrected logic would have produced from scratch. Full replay sidesteps all of that: it doesn't matter how the projection got into a bad state, because the bad state is discarded and rebuilt from ground truth. This is also why immutability of the event log matters so much in event sourcing - it's precisely what makes 'just replay it' a reliable recovery strategy rather than a leap of faith. ## What the handlers must guarantee For a rebuild to actually produce correct results, the handlers must satisfy two properties. 1. **First, determinism**: given the same sequence of events in the same order, the handler must always produce the same resulting state. If a handler's logic depends on anything outside the event data itself - the current wall-clock time, a call to an external pricing service whose answer may have changed, a random ID generator - then a rebuild months later can produce different results than the original run did, and the 'fix' may itself be subtly wrong or non-reproducible. 2. **Second, and often overlooked**, handlers must be free of externally visible side effects that shouldn't be repeated: if the original event processing also sent a 'welcome' email or fired a webhook to a partner system, a naive full replay through those same handlers would resend every email and webhook ever triggered, which is a serious operational hazard. In practice, teams handle this either by keeping notification and side-effect logic in entirely separate handlers from pure state-projection handlers (so a rebuild only re-runs the latter), or by running rebuilds with external effects explicitly suppressed. ## The trade-off The trade-off is mostly about time and availability versus correctness confidence. Rebuilding is conceptually simple and trustworthy, but for a large event log it can be slow - hours for millions of events - during which either the projection is unavailable (if rebuilt in place) or a duplicate resource cost is paid (if rebuilt alongside the old version for a blue-green cutover). It also assumes the fix is actually correct; a rebuild with a still-wrong handler just produces a different wrong answer, faster than a hand-written migration would have, but wrong all the same, so rebuilds are usually validated against a sample or a shadow-traffic comparison before cutover. ## Failure modes in practice Failure modes in practice include: - forgetting to suppress side-effecting handlers during rebuild, which floods customers with historical notifications - non-deterministic handlers producing a rebuild that doesn't match expectations and requires a second investigation to figure out why - rebuilding in place without a blue-green swap, which takes the read API offline or serves partial data mid-rebuild A well-known concrete pattern here is the **projection-versioning** approach used by frameworks like Axon Framework, where a new projection version is built alongside the old one under a new name or table, validated, and only then does a router or feature flag cut reads over to it - avoiding any window where reads see either downtime or a half-rebuilt table.