What's the difference between a 'live' subscription and a 'catch-up' subscription when a projection consumes an event stream, and why would a projection need both?

## The two modes of subscription An **event stream**, in this context, is an ordered sequence of events (per topic, per aggregate, or per some other grouping) that a projection reads and processes. A **subscription** is the mechanism a projection handler uses to receive events from that stream. Two distinct modes of subscription show up across event-driven systems: - **catch-up**, meant for reading a backlog of already-existing events - **live**, meant for receiving newly-appended events as they happen ## How each one behaves Mechanically, the split looks like this: | Mode | Behaviour | |---|---| | **Catch-up** | starts at a specific position — the very beginning of the stream, or a checkpoint the projection previously saved — and reads sequentially forward through everything that already exists, typically in batches, optimized for throughput rather than latency, until it reaches whatever was the 'current head' of the stream when it started | | **Live** | by contrast, is optimized for low latency: as soon as a new event is appended, it's pushed (or nearly-immediately polled) to the subscriber, so the consumer learns about it within milliseconds rather than waiting for a bulk-read cycle | Concretely, systems like EventStoreDB expose a live subscription API that pushes new writes to connected subscribers, and change-data-capture systems like MongoDB Change Streams push document changes as they commit. ## Why one projection needs both Both modes exist because a projection has two very different jobs at different points in its life. 1. A brand-new projection, or one being rebuilt after a bug fix, has to process potentially millions of historical events before it's useful at all — doing that through a low-latency, one-at-a-time push path would be needlessly slow and wasteful of connection/notification overhead, so a bulk, throughput-optimized catch-up read is the right tool. 2. Once a projection has processed everything that already existed, though, continuing to run bulk reads on a polling interval to look for new data is wasteful and adds unnecessary latency compared to being pushed new events as they arrive — so mature event-store client libraries typically expose a single subscription abstraction that internally starts in catch-up mode and transitions to live mode automatically once it reaches the tail of the stream. ## The trade-off The trade-off is exactly that split of concerns: - **Catch-up mode favors throughput** at the cost of freshness, which is perfectly fine because during a rebuild or initial load, staleness relative to 'right now' is already expected and accepted. - **Live mode favors freshness** at the cost of added coordination complexity, particularly around the moment of transition between the two. Some simpler systems skip this elegance entirely and just poll on a fixed short interval for everything, trading a bit of latency for a lot less implementation complexity — this is a legitimate and common simplification, especially with Kafka, where a single continuous consumer poll loop naturally reads old records and then new ones without the application needing to think of them as separate modes at all. ## Where it breaks 1. **The handoff itself** is the classic failure mode. If a projection only starts its live listener after fully finishing catch-up, any events appended to the stream during the time catch-up was still running — after it captured its notion of 'current head' but before the live listener actually started — can be missed entirely. The safe pattern is to start buffering live events before or during catch-up, finish catch-up, then drain the buffer while deduplicating by event position so nothing appended during the overlap is skipped or double-applied. 2. **A second, related failure** is catch-up resuming from a stale or incorrect checkpoint after a restart — if the last-processed position wasn't durably persisted before a crash, catch-up either reprocesses a chunk of already-applied events (safe only if the handler is idempotent) or, worse, skips ahead past events that were never actually applied. 3. **A third failure** is running a large catch-up read against the same infrastructure and connection budget as live production traffic, which can starve production reads or writes of I/O capacity while the rebuild churns through history. ## How real systems expose it - **EventStoreDB's catch-up subscription API** is a direct, named example of this pattern: it reads from a given position through to the live end of the stream and then keeps delivering new events without the caller having to manage the transition explicitly. - **MongoDB Change Streams** offer an analogous idea via resume tokens: a consumer can catch up on everything since a given token and then continue receiving live changes. - **Kafka**, notably, doesn't expose these as two separate APIs at all — a consumer group simply starts reading from a stored offset (or 'earliest') and naturally proceeds toward the tail as one continuous poll loop, which is exactly why Kafka-based projections tend to be simpler to reason about on this specific point than systems with an explicit dual-mode subscription API.