A platform team decides to fix a bug in a widely-used read-model projection by triggering a full replay of every aggregate's event stream to rebuild it from scratch across the whole system. What can go wrong at scale, and how would you design the rebuild to avoid taking down the system or producing a projection that's subtly different from what live processing would have produced?

## A different operation in kind Rehydrating a single aggregate is a small, local operation — fold a few hundred or thousand events, done in milliseconds. Rebuilding a shared read-model projection by replaying every aggregate's stream in the system is a fundamentally different operation in kind, not just degree, and the failure modes that show up only appear at this scale. ## The replay storm The first and most immediate risk is what's often informally called a **"replay storm."** A projection rebuild has to read every event for every aggregate the projection cares about — potentially millions of events across thousands or millions of streams — in a relatively short window, because the business pressure is usually "fix the bug now." That read load lands on: - the same event store infrastructure that live command handling depends on for its own rehydration reads; - and any downstream systems the projector's apply logic calls out to (search indexes, caches, other services via API). Without deliberate throttling, this read burst competes with live traffic for the same database connections, I/O, and CPU, and can measurably degrade or outright take down normal operation — the fix for a projection bug becomes an incident in its own right. The standard mitigation is treating a full rebuild as a first-class, rate-limited operation: reading streams in controlled batches, spreading work across time or partitions, and, where the store supports it, using a dedicated read replica or a change-data-capture/log-based export rather than hitting the same primary that live traffic uses. ## Non-determinism, which announces nothing The second risk is subtler and more dangerous because it doesn't announce itself with an outage: **non-determinism** producing a rebuilt projection that's quietly different from what live, incremental processing would have produced. This happens whenever apply or projection logic isn't a pure function of (state, event) — if it: - reads the current wall clock instead of a timestamp stored on the event; - calls an external service whose answer can change over time; - depends on the arrival order of events across different aggregates (rather than being correctly scoped to per-aggregate ordering); - or has any hidden dependency on execution context. In live production, that logic executed once, at a specific real moment, and its output became part of the historical record indirectly (through side tables, caches, or derived events). During a replay months or years later, the same logic executes again but the world it's now running in has moved on — external answers changed, clocks are different, related aggregates may have since evolved — so the rebuilt projection can differ from the original in ways that are individually small but collectively meaningful, and because both versions look internally consistent, nobody notices until a downstream report or customer disagrees with a number. ## What consumers see mid-rebuild A third risk is what the rebuilt projection's consumers see while the rebuild is in progress. If the rebuild writes directly into the same table or index the live projection already serves reads from, readers see a mix of old and new data mid-rebuild — some rows rebuilt, some not yet touched — which is itself an inconsistent, confusing state that's worse than either the old buggy version or the new fixed version taken as a whole. The standard architectural answer is to build the new projection into an entirely separate store or table (a **"blue/green"** or **shadow** projection), then: 1. Let it fully catch up, including the events that continue arriving live while the rebuild is running. 2. Validate it — spot-check a sample of aggregates against the currently-live projection, or against independently recomputed values. 3. Only then atomically switch reads over to the new version — after which the old one can be torn down. This also naturally solves the "replay caught up to a moving target" problem: because live events keep arriving during a long rebuild, the rebuild process has to be designed to consume both the historical backlog and the ongoing live stream, converging on "caught up" rather than assuming a fixed, static amount of work. ## Ordering guarantees Ordering guarantees are the fourth, more architecture-specific concern. Event sourcing's correctness generally only requires strict ordering within a single aggregate's stream, not a global total order across all aggregates — but some projections (a cross-aggregate report, a materialized join) implicitly depend on processing order across aggregates in ways that aren't obvious until a replay reorders things differently than live processing happened to. On a system built over Kafka, for instance, per-partition ordering is guaranteed but cross-partition ordering is not, so if a projection was accidentally written assuming a global order that live traffic happened to mostly preserve (because throughput was low), a fast, parallelized replay that processes many partitions concurrently can expose the bug by producing genuinely different interleavings. ## Where it shows up A concrete real-world example: teams running projections on top of **Kafka Streams** or **ksqlDB** regularly hit exactly this class of problem when doing a full topic replay to fix a stateful aggregation bug — they mitigate it with bounded consumer group rebalancing, dedicated rebuild consumer groups reading from offset zero into a separate output topic/table, and a documented cutover step, rather than replaying in place against the topic the live application already reads from.