An upload to a single S3 bucket must trigger three independent consumers: a thumbnailer, an antivirus scan, and an audit writer. How do you wire that with S3 event notifications, and what does routing through EventBridge change compared with S3's native destinations?
answer
- one event type cannot match twice
- you need a fan-out hop
- a topic or a bus in front
- each consumer keeps its own backlog
- replay needs an archive
basics
~20 sS3 refuses overlapping notification configurations for one event type, so three consumers on the same prefix need a fan-out point: either one SNS topic with three subscribers, or EventBridge notifications with three independently filtered rules. EventBridge adds richer filtering, archive and replay, and cross-account routing.
solid answer
~50 sYou cannot point three native configurations at the same event type and prefix — S3 rejects overlapping filters — so you need one fan-out hop. The classic shape is S3 to an SNS topic, with an SQS queue per consumer subscribed to it; each consumer then owns its own backlog, retries and dead-letter path, and a slow antivirus scanner cannot stall the thumbnailer. The modern shape is to enable EventBridge notifications on the bucket and write three rules on the default bus, each with its own event pattern and target. EventBridge buys content-based filtering on any field, many rules matching the same event, archive and replay, and cross-account or cross-region routing; it costs an extra hop of latency and a second service to reason about. Either way I would still land each consumer behind a queue rather than invoking business logic directly.
go deeper
Know the four destinations an S3 bucket can notify — Lambda, SQS, SNS and EventBridge — and that a single event type on one prefix cannot be pointed at several destinations at once.
Explain why the overlap restriction forces a fan-out hop, and describe the SNS-to-many-queues shape and the EventBridge-rules shape concretely enough to draw them.
Argue the isolation case: per-consumer queues, independent retries and dead-letter paths, and the destination resource policy that makes the wiring actually work. Name what EventBridge adds and what the extra hop costs.
Own the standard for the organisation — whether new buckets default to EventBridge so consumers can be added without touching the producer side, and how central teams subscribe to events from accounts they do not control.
## Why one bucket cannot simply list three functions S3's notification configuration allows several entries, but it rejects two entries that share an event type and whose key filters overlap. Three consumers all interested in `s3:ObjectCreated:*` under `uploads/` are the definition of overlap, so the native configuration cannot express the requirement. That constraint is the whole reason fan-out is a design question at all. ## Option 1: SNS as the fan-out point S3 publishes to a single SNS topic. Each consumer subscribes — in practice, an SQS queue per consumer, with the worker reading its own queue. ``` S3 -> SNS topic -> SQS (thumbnails) -> worker -> SQS (av-scan) -> worker -> SQS (audit) -> worker ``` What this buys you: - **Isolation.** Each consumer has its own backlog. If the antivirus scanner is down for an hour, its queue grows while thumbnails keep flowing. - **Per-consumer failure handling.** Retries and a dead-letter path belong to each queue, not to the shared notification. - **Backpressure.** A burst of ten thousand uploads becomes queue depth rather than ten thousand simultaneous invocations. The cost is three more resources to own and a topic whose subscription list is the real fan-out map — easy to forget when someone later asks "what runs when a file lands here?" ## Option 2: EventBridge notifications Enable EventBridge on the bucket. S3 then publishes every event to the account's default event bus, and you write one rule per consumer. Rules match independently, so five rules matching one event is normal rather than an error. What EventBridge adds over the native path: - **Content-based filtering.** A rule's event pattern can match on any field in the event, not just a key prefix and suffix — the specific event name, the object size, anything-but conditions. - **Many targets, many rules.** The overlap restriction disappears entirely; adding a fourth consumer is a new rule, not a re-plumbing. - **Archive and replay.** You can archive matching events and replay them later, which is the only clean way to re-run a pipeline over a window of past notifications. - **Cross-account and cross-region routing.** Events can be forwarded to a bus in another account, which is how a central security team subscribes to buckets it does not own. What it costs: one more hop, so slightly more latency; a second service in the failure path; and events for the bucket that now exist whether or not any rule matches them. ## What does not change Both paths inherit the same underlying semantics. Delivery is still at-least-once, so every consumer must be idempotent. Neither path backfills objects that existed before the configuration. And in both cases the *destination* must permit S3 to publish to it: an SNS topic policy or SQS queue policy granting the `s3.amazonaws.com` service principal, scoped with `aws:SourceArn` for the bucket and `aws:SourceAccount` for your account so a stranger's bucket cannot publish into your topic. A missing or over-broad destination policy is the single most common reason a freshly wired notification appears to do nothing. ## Choosing between them ``` One consumer, simple key convention -> native destination (SQS or Lambda) Several consumers, same events -> EventBridge rules, or SNS + SQS Filtering on something other than the key -> EventBridge Need replay of past notifications -> EventBridge with an archive Events must reach another account -> EventBridge cross-account bus Strict low latency, one hop only -> native destination ``` In a new build I default to EventBridge when there is any chance of a second consumer, because retrofitting fan-out onto a native configuration means changing the S3 side while producers are live. When the pipeline is genuinely one bucket to one queue and always will be, the native configuration is one fewer thing to explain. ## The shape to describe out loud "S3 cannot fan out by itself, so I put one fan-out point immediately after it, and I give every consumer its own queue so their failures are independent." Then name which fan-out point and why. Interviewers are listening for the isolation argument at least as much as for the service names.
- Why put an SQS queue in front of each consumer rather than subscribing the Lambda functions to the topic directly?A queue gives each consumer a durable backlog it owns. Bursts become queue depth instead of a concurrency spike, a consumer that is failing retries against its own dead-letter path without affecting siblings, and you can pause one consumer by stopping its poller while events keep accumulating safely. Subscribing functions directly couples every consumer's availability to the moment the event fires.
- The notification is configured and uploads are happening, but the SQS queue stays empty. What do you check first?The queue's resource policy. S3 publishes as the service principal s3.amazonaws.com, and the queue policy must allow sqs:SendMessage for it, normally conditioned on aws:SourceArn for the bucket and aws:SourceAccount. Missing or mis-scoped destination policies are the usual cause of silently dropped notifications. After that, check the filter actually matches the keys being written and that the queue is in the same Region as the bucket.
- When would you skip event notifications altogether and have consumers list the bucket on a schedule?When the workload is genuinely batch and completeness matters more than latency — a nightly reconciliation, for example. Listing is a positive check that catches anything a lost or misconfigured notification missed, whereas notifications are a push you cannot audit after the fact. Many mature pipelines run both: notifications for the fast path, a periodic sweep as the safety net.
saying these in an interview costs you the question
- Registers three overlapping configurations and expects all to fire
- Treats SNS and SQS as interchangeable for fan-out
- Forgets the destination's resource policy entirely
- Claims EventBridge makes delivery exactly-once
- Invokes business logic directly with no queue for backpressure