Messaging Foundations

Without a fix, the broker keeps redelivering the bad message forever, and it can jam up processing for everyone behind it. A DLQ routes a message elsewhere automatically after it fails too many times, so the queue keeps moving and someone can look at the bad message later.

Some brokers throw a message away as soon as it's been successfully handled once. Others keep every message around for a set amount of time no matter who's read it, so you can go back and re-read old messages later, at the cost of using more storage.

Queue systems like SQS or RabbitMQ can automatically move a failed message to a separate DLQ for you. Kafka has no such built-in feature — the consumer's own code has to catch the failure and manually publish the message to a separate 'DLQ topic' itself.

Adding a number to a running total isn't safe to repeat; do it twice and the total is wrong twice. Idempotent updates instead set a value to its final state, like 'balance is now X', or use a unique key so replaying the same update lands on the exact same result every time.

With a queue, you need three separate queues (or a fan-out router) so each service gets its own copy, because one message can only be taken once. With a log, all three can just read the same shared stream independently, each keeping its own place.

A rebalance is when the group reshuffles which consumer reads which partition (e.g., because someone joined or left). During it, processing pauses briefly, and a partition can move to a different consumer that has no memory of what was being processed, so you can see duplicate or stalled processing, though the log order itself isn't changed.

Multiple workers grabbing jobs from one shared line means a job's owner might crash mid-task, so the system needs a way to notice and give that job to someone else — otherwise it's silently lost or, if handled sloppily, done twice.

The publisher doesn't send directly to a queue - it sends to a routing component that reads a label on the message and copies it into whichever queues match that label. If nothing matches, the message is usually just dropped unless the broker is told to save unmatched messages somewhere.

If a message failure means something urgent needs to happen right away (like a fraud alert) or reprocessing it later could cause harm (like applying a stale price), quietly parking it in a DLQ for someone to check later isn't good enough — you need a faster, more active response instead.

Kafka can bundle writing output records and committing the input offset into one all-or-nothing transaction, using a producer ID and sequence numbers to avoid duplicate writes on retry, so a crash mid-write either commits everything or nothing — but only within Kafka.

Stateful operators keep a local, on-disk 'memory' (like a running total) next to each processing task, and also write every change to a durable, replayable backup log. If the task dies or moves machines, it rebuilds its memory by replaying that backup log.

A watermark is the stream processor's best guess of 'we won't see any more events older than this timestamp.' Once the watermark passes a window's end time, the system treats that window as closed and emits its result, accepting it might occasionally be wrong if a late straggler shows up after.

A separate 'have I seen this before' table costs extra storage, a database write on every message, and upkeep to stop it growing forever. If the actual operation is already safe to repeat on its own, like setting a status to a fixed value, that extra table isn't needed at all.

A queue - you want the task done once by one worker and then forgotten, with automatic retry if it fails. A log is overkill here because nobody else needs to replay or re-read 'send this email' later; you'd actually want to avoid accidentally resending it.

That one heavy user's events all pile onto a single partition no matter how many partitions the topic has, so that one partition gets overloaded and lags behind while the rest stay fine. Fixes usually involve splitting that user's events across a few partitions and reordering them again on the consumer side, or treating that user as a special case.

Push means the broker decides when to send you the next message, which risks flooding a slow consumer. Pull means the consumer asks for more only when it's ready, so it naturally never asks for more than it can handle.

In a queue, once one consumer finishes a message, it's normally deleted - gone for good. In pub-sub, each subscriber has its own independent copy and cursor, so one subscriber finishing doesn't remove the message for the others, and depending on the system it may still be replayable later.

Schema compatibility checks only verify the message's structure is technically valid. Contract testing goes further and checks that consumers get the actual data and values they depend on, catching cases where a message is structurally fine but breaks business logic downstream.

FULL mode is the safest but the most restrictive: it forces every schema change to be additive with defaults, which slows down teams that need to rename fields, tighten types, or make big changes. For internal, tightly-coordinated systems or truly transient data, a looser policy or skipping the registry entirely can be the more pragmatic choice.

A broker is a separate service that sits in the middle and stores messages for you. A brokerless library like ZeroMQ instead makes the sending and receiving programs talk straight to each other over the network, which is faster and has one less thing to run, but now each program has to handle things the broker used to handle, like remembering where everyone is and what to do if a message can't be delivered yet.

Different parts of the pipeline need different guarantees based on how costly a duplicate or a loss is there — you can't just pick one setting for the whole system. Use real exactly-once only where it's cheap, like Kafka-to-Kafka, and layer at-least-once plus deduplication everywhere the guarantee has to reach outside Kafka, matching the mechanism to each sink's actual risk and cost.

For OrderPlaced, use pub-sub - every interested service (inventory, email, analytics) needs its own full copy at its own speed. For CapturePayment, use a point-to-point queue so only one worker handles each payment exactly once, with per-account ordering.

You look at what kind of error occurred — a 'try again later' error (like a timeout) should get retried, but a 'this will never work' error (like bad data) should skip straight to the DLQ instead of burning through retries pointlessly.

The system bundles 'I read this event,' 'I updated my running total,' and 'I wrote the result' into one all-or-nothing operation using transactions, so a crash can never leave things half-done - either all three happened or none did.

One shared 'have we seen this before' table gets hammered by every single message across a huge system, becomes a bottleneck, and keeps growing forever. At large scale, teams split it up, for example one store per partition, or use a fast key-value store with automatic expiry instead of one giant table everyone reads and writes.

In Kafka, that one bad message can jam up the whole line behind it, because the consumer must go through messages in order - everything after gets stuck waiting. In SQS, a bad message just gets set aside (retried, then dead-lettered) while all the other messages keep moving normally.

You either force everything through one lane (slow but simple), or you let events flow through many lanes and later merge them back into one order using timestamps or a sequence number handed out by a single central 'ticket booth,' accepting some extra delay or complexity to reconstruct that single true order.

When lots of teams share one messaging system, one team's slow consumer can hurt everyone else if it's allowed to fill up shared storage or overload shared brokers. The fix is isolating teams from each other (quotas, per-topic limits) plus giving each team clear tools (autoscaling, dead-letter queues, shedding) to fix their own lag before it spreads.

All three can express similar rules for adding/removing fields safely, but they differ in how evolution is tracked: Avro relies on a full schema document and reader/writer resolution, Protobuf uses stable numbered field tags baked into the format itself, and JSON Schema is more flexible but has weaker native tooling for enforcing evolution rules. The right pick depends on your ecosystem, tooling, and how much schema strictness you actually want.