skip to content

What is Amazon EventBridge Pipes, and when does it replace a glue Lambda function written to move events from one AWS service to another?

level: middleimportance: should knowfreq 50%

answer

  1. managed connector, not a bus
  2. one source, one target
  3. four stages in a fixed order
  4. filter before you pay for compute
  5. the plumbing function disappears

basics

~20 s

EventBridge Pipes is a managed point-to-point connector: one source (SQS, Kinesis, DynamoDB Streams, MSK, Amazon MQ) to one target, with optional filtering and enrichment in between. It replaces glue Lambda code whose only job is polling, filtering and forwarding.

solid answer

~50 s

A pipe has four stages: a **source** it polls, an optional **filter** using EventBridge event-pattern syntax, an optional **enrichment** call, and a single **target** with an optional input transformer. Sources are the pollable ones — SQS, Kinesis Data Streams, DynamoDB Streams, Amazon MQ, and Kafka (MSK or self-managed). Enrichment can be a Lambda function, an API destination, API Gateway, or a synchronous Step Functions Express workflow. I reach for Pipes when the code I would otherwise write is pure plumbing — poll a queue, drop the records I don't care about, reshape the payload, call the next service. That code disappears, along with its cold starts, retries and deployment. I keep a Lambda when the step has real business logic, needs libraries, or has to fan out, because a pipe delivers to exactly one target.

code

json · 6 lines
json
{
  "body": {
    "status": ["PLACED"],
    "orderTotal": [{ "numeric": [">", 100] }]
  }
}

go deeper

for a junior

Know that EventBridge Pipes connects one source to one target and can filter and reshape events without you writing a Lambda. Be able to name a couple of sources, such as SQS and DynamoDB Streams.

for a middle

Explain the source, filter, enrichment, target sequence, which services are legal at each stage, and why filtering before enrichment saves both latency and invocation cost. Say plainly when you would still write a function.

for a senior

Show judgment about migrating existing glue functions: what the pipe inherits from the source's failure handling, what the single IAM role must allow, and what visibility you gain or lose compared with code you instrumented yourself.

for a principal

Own the boundary between configuration and code across a fleet. Argue when eliminating glue functions genuinely reduces operational surface versus when it scatters logic into console-managed configuration that nobody reviews or tests.

## What a pipe is EventBridge Pipes is a feature of Amazon EventBridge that creates a **point-to-point** connection between one event producer and one consumer. It is deliberately not a bus: there is no pattern-matched fan-out, no multiple targets, no archive. One pipe = one source, one target. That narrowness is the point — it makes Pipes a drop-in replacement for the small, boring integration functions that accumulate in every event-driven account. ## The four stages Every pipe runs the same pipeline: 1. **Source** — the pipe polls it on your behalf, using an IAM role you supply. Sources are the *pollable* AWS event producers: Amazon SQS, Kinesis Data Streams, DynamoDB Streams, Amazon MQ (ActiveMQ and RabbitMQ), Amazon MSK, and self-managed Apache Kafka. Batch size and a batching window are configurable, and for ordered sources the pipe preserves order within a shard or message group. 2. **Filtering** (optional) — the pipe evaluates each event against one or more event patterns using the same matching syntax EventBridge rules use, so `prefix`, `numeric`, `anything-but` and friends all work. Events that do not match are discarded before enrichment, so they never reach your code. 3. **Enrichment** (optional) — a *synchronous* call that transforms or augments the event. The allowed enrichment services are Lambda, EventBridge API destinations, Amazon API Gateway, and AWS Step Functions **Express** workflows (Standard workflows are asynchronous, so they cannot be an enrichment). The response replaces the event payload passed to the target. 4. **Target** — one target, invoked with the enriched payload. The target list mirrors EventBridge's, and a target input transformer can reshape the payload one last time without any code. ```json { "body": { "status": ["PLACED"], "orderTotal": [{ "numeric": [">", 100] }] } } ``` That filter, applied to an SQS source, forwards only placed orders over 100 — the `if` statement you would otherwise have written and tested. ## Why this replaces glue code The classic glue Lambda does four things: it is wired to a source by an event source mapping, it deserializes the batch, it drops uninteresting records, and it calls another service's SDK. None of that is business logic, but all of it is code you own — packaged, deployed, permissioned, monitored, and paged on. A pipe expresses the same intent as configuration. Concretely you stop maintaining: the SDK client and its retry configuration, the batch loop, the filter conditions, the payload mapping, and the function's own error handling. Secondary benefits matter in interviews: filtering happens before your enrichment runs, so you are not invoking (and paying for) a function that immediately returns; the pipe emits its own CloudWatch metrics; and Pipes supports execution logging to CloudWatch Logs, S3 or Firehose at OFF/ERROR/INFO/TRACE levels, which gives you per-event visibility that a hand-rolled function only has if someone wrote the log lines. ## When a Lambda is still the right answer - **Fan-out.** A pipe has exactly one target. If the same event must reach three consumers, publish to an SNS topic or a custom event bus instead — or make the bus the pipe's target. - **Real logic.** If the step makes decisions, calls several services, or needs a library, that is a function, not plumbing. It can still *be* the pipe's enrichment or target. - **Non-pollable sources.** Pipes sources are the polling integrations. An S3 event notification or a custom `PutEvents` payload arrives on a bus, not through a pipe. - **Long-running work.** Enrichment is synchronous and in the request path; a slow enrichment slows the whole pipe. ## The operational picture A pipe has one IAM role that must permit reading the source (for example `sqs:ReceiveMessage`, `sqs:DeleteMessage`), invoking the enrichment, and invoking the target — a common first failure is a role that can read but not deliver. Failure handling stays source-shaped: for an SQS source, a message that keeps failing goes to the queue's own dead-letter queue through its redrive policy; for stream sources, the pipe itself carries retry and dead-letter settings. That inherited behaviour is exactly what a glue Lambda's event source mapping did, which is why the migration is usually a like-for-like swap.

  • Can a pipe deliver the same event to more than one target?
    No — a pipe is point-to-point and has exactly one target. If you need fan-out, make the target an SNS topic or a custom EventBridge bus and let that do the multiplexing, or create a second pipe over the same source if the source supports multiple independent consumers.
  • Which Step Functions workflow type can be used as a pipe enrichment, and why?
    Only an Express workflow. Enrichment is a synchronous call whose response becomes the payload sent to the target, and Express workflows can be invoked synchronously. A Standard workflow starts asynchronously and returns an execution ARN, so it can be a pipe *target* but not an enrichment.
  • Where in the pipe would you reshape a payload the target cannot accept?
    Two places. The enrichment stage if the reshaping needs data you must fetch or compute; the target input transformer if it is a pure remapping of fields already present. Prefer the input transformer — it is configuration, costs nothing extra, and adds no latency.

A bus is a post office that copies one letter to every subscriber; a pipe is a single conveyor belt with a sorter and a labelling machine bolted on partway along.

saying these in an interview costs you the question

  • Says a pipe can fan out to several targets at once
  • Thinks Pipes replaces the event bus entirely
  • Claims any AWS service can be a pipe source
  • Believes enrichment can be a Standard Step Functions workflow
  • Assumes Pipes needs no IAM role because it is managed

context

resolving links…