In Kinesis Data Streams, how does an enhanced fan-out consumer differ from the default shared-throughput consumer, and when is the extra cost justified?

## The default: shared throughput A standard Kinesis Data Streams consumer gets a shard iterator and calls `GetRecords` in a loop. The shard grants **2 MB per second of reads and at most 5 `GetRecords` calls per second — in total, across every standard consumer reading that shard**. That budget is not per application. Consequences follow directly: - With one consumer, you comfortably out-read a shard that ingests at most 1 MB/s. Life is good. - With two, each averages about 1 MB/s and they begin competing for the five calls per second. - With three or more, throttling (`ReadProvisionedThroughputExceeded`) becomes routine, and every consumer's `GetRecords.IteratorAgeMilliseconds` starts to climb. A batch job doing a big backfill can starve a real-time consumer that was fine yesterday. Latency is the second cost. Polling means a record waits for the next poll; with one consumer polling at the maximum rate you land around 200 ms of propagation delay, and with several consumers sharing five calls per second each application must poll less often, so it gets worse the more consumers you add. ## Enhanced fan-out: dedicated throughput and push delivery Enhanced fan-out (EFO) changes two things at once. **Allocation.** You call `RegisterStreamConsumer` to create a named consumer on the stream, which gets its own ARN. That consumer receives a **dedicated 2 MB/s per shard**, independent of every other consumer. Ten registered consumers on a 4-shard stream each read up to 2 MB/s per shard; nobody's backfill starves anyone's alerting pipeline. **Transport.** Instead of polling `GetRecords`, the consumer opens a `SubscribeToShard` call — an HTTP/2 stream over which Kinesis **pushes** records as they arrive. Typical propagation is about 70 ms and does not degrade as you add consumers. The subscription is not permanent: a `SubscribeToShard` call lasts up to five minutes, after which the client renews it. Any real client — the KCL, the Lambda event source mapping in EFO mode, the Flink Kinesis connector — handles that renewal for you. ```bash aws kinesis register-stream-consumer \ --stream-arn arn:aws:kinesis:eu-west-1:111122223333:stream/events \ --consumer-name realtime-alerts aws kinesis list-stream-consumers \ --stream-arn arn:aws:kinesis:eu-west-1:111122223333:stream/events ``` ## What it costs EFO is billed on two dimensions: a charge per **consumer-shard-hour** (registered consumers multiplied by shards multiplied by hours) and a charge per **GB of data retrieved**. The first is the one that surprises people: the meter runs on shard count and consumer count regardless of how much data you actually read, so registering five consumers on a 64-shard stream is a standing cost even overnight. Standard consumers, in provisioned capacity mode, add no incremental charge at all — you already pay for the shard. There is also a default quota on the number of consumers registered for EFO per stream (a small number, raisable through a service-quota request). If your architecture calls for dozens of independent readers, that quota is a design constraint worth checking early. ## Choosing Reach for EFO when: - **Multiple applications read the same stream** and must not interfere with each other. This is the dominant reason. - **One consumer is latency-critical.** Roughly 70 ms versus 200 ms-plus matters for alerting, fraud scoring, or live personalisation; for a five-minute batch aggregate it does not. - **A backfill must run alongside production.** Registering the backfill as its own EFO consumer isolates its aggressive reads from the live path. Stay with standard consumers when there is one reader, or two that both tolerate polling latency, and the stream is wide enough that the per-consumer-shard-hour charge would dominate. A useful middle position: not every consumer has to make the same choice. Register the latency-sensitive application for EFO and leave the analytics job on shared polling — the analytics job then has the shared 2 MB/s largely to itself. ## Things candidates get wrong EFO does not increase **write** capacity, and it does not increase the shard's storage or retention. It also does not fix a hot shard: if one shard receives most of the traffic, a dedicated 2 MB/s on that shard may still not keep up with what the producers wish they could write, and the shard's 1 MB/s write ceiling is untouched. Finally, EFO is a per-consumer property, not a stream-wide mode — you register consumers individually, and a stream can serve both kinds at once.