Messaging Foundations

A DLQ is a separate queue where messages that fail processing repeatedly get moved to, instead of blocking the main queue or being retried forever, so the app keeps working while someone looks at the bad message later.

At-most-once can lose a message but never repeats it. At-least-once never loses a message but can repeat it. Exactly-once means every message is processed once, no loss and no duplicates. Most real systems use at-least-once plus deduplication.

An offset is a bookmark marking how far a consumer has read in a partition. Since the log isn't deleted after reading, moving the bookmark backward lets the consumer re-read and reprocess old events.

Idempotent means doing something twice has the same effect as doing it once. The consumer remembers which messages it already handled (by ID) and skips reprocessing the effect, like charging money again, even if the message arrives more than once.

In a queue, once a message is picked up and confirmed done, it's deleted - gone for good. In an event log like Kafka, the message stays stored so other readers can still see it later, or the same reader can go back and read it again.

Each partition is its own ordered line of messages, like a single queue. Different partitions are separate queues running independently, so messages from different partitions can arrive in any order relative to each other.

A producer drops a message off at a broker (like a queue) without knowing who reads it; a consumer picks messages up without knowing who sent them. Each side can change, restart, or scale on its own without breaking the other.

In pub-sub, every subscriber gets its own copy of each message (fan-out, one-to-many). In a queue, each message goes to exactly one consumer picked from the pool (point-to-point, one-to-one), even if many workers are listening.

A schema registry is a shared service that stores the agreed-upon 'shape' of each message type. Producers register a schema and get a short ID; consumers use that ID to look up the schema and read the message correctly, instead of sending the whole schema every time.

A queue delivers each message to exactly one consumer, like a shared to-do list where each task is picked up once. A topic broadcasts each message to every subscriber, like a radio station everyone tuned in hears.

You wait a bit longer between each retry (backoff) instead of retrying instantly, and you cap the total number of tries (threshold) so a broken message eventually gets pulled aside into the DLQ instead of hammering the system forever.

Committing the offset once per batch is faster but replays the whole batch if a crash happens partway through, since Kafka only tracks one checkpoint per partition, not per message. Committing after every single message shrinks that replay window but is much slower.

Compaction keeps only the newest record for each key and throws away older ones, instead of deleting everything older than some age. It's like keeping only the latest version of each row in a table, not a full history.

A tumbling window is a fixed time slice that never overlaps the next one, like every 5 minutes. A sliding window overlaps and updates continuously, like a 'last 5 minutes' view refreshed often. A session window has no fixed size - it groups events until there's a gap of inactivity.

A table that lists 'already handled' message IDs can still let two duplicates through if both are checked at the same instant before either writes its ID down, like two people checking an empty sign-up sheet at once and both writing their name first. Fix it by making the 'write' step atomic, for example a database unique constraint.

A log consumer remembers a number (an offset) marking its place in an ordered list of messages, and moves that number forward as it reads. A queue consumer instead tells the broker 'I'm done with this one specific message,' and the broker deletes just that message.

The system picks a partition based on a formula applied to a 'key' you attach to each message. Use the same identifying value (like the entity's ID) as the key every time, so all its events always land in the same partition and stay in order.

The topic's data is split into chunks (partitions), and the broker hands each chunk to exactly one instance in the group. So the group as a whole reads everything once, but each message is only handled by one member.

The broker assumes the message wasn't handled and gives it to another consumer after a timeout, so the message isn't lost. But this means the same message might get processed twice - that's called at-least-once delivery.

The broker hands each message to whichever available consumer is free (round-robin or pull-based), so work spreads across consumers. Once messages are spread across multiple consumers, you generally lose the guarantee that they're processed in the exact order they were sent.

BACKWARD compatibility means new schema versions can still be read using the code built for the old schema. FORWARD is the opposite: old code can read data written with the new schema. FULL means both directions work at once.

The producer's serializer sticks a magic byte and a 4-byte number (the schema's ID) at the very start of the message, followed by the actual encoded data. The consumer reads those first 5 bytes to know which schema to fetch, then uses that schema to decode the rest.

Splitting a topic into partitions lets multiple machines share the load, but messages are only guaranteed to arrive in order within a single partition, not across the whole topic. Which partition a message lands in is usually decided by a key you choose, like a customer ID.

You need alarms that page someone when the DLQ starts filling up, and a safe process to review each stuck message, fix whatever caused it (or confirm the world hasn't changed), and only then resend it back to be processed — never just blindly replaying everything.

Because 'exactly-once' guarantees usually only cover the messaging system itself. The moment a message triggers an outside effect — an email, an API call, a non-transactional database write — that effect can still happen more than once, so the real guarantee is 'looks like exactly once if downstream systems are built to tolerate replay.'

If the sender retries and the queue gives the retry a brand-new message ID, checking 'have I seen this exact message ID' won't catch it as a duplicate. Instead, dedupe on something the business itself considers the same thing every time, like the order's own ID, so retries collapse together no matter what ID the broker assigns.

With a log, you can rewind your reader to before the bug started and reprocess those 3 days from the original messages, since they're all still stored. With a queue, those messages are already deleted once the first, buggy pass consumed them, so there's nothing left to replay - you'd need another way to get that data back.

More partitions means more independent lines that can be read and written in parallel, so more machines can work at once and throughput scales up. But you can never guarantee that the whole topic's events arrive in one single overall order - only within each line - and resizing can even split up an entity's history across old and new lines.

If readers can't keep up with writers, unprocessed work piles up. Eventually old messages get deleted before anyone reads them, or storage fills up, or downstream data goes stale. Back-pressure means slowing the producer down, scaling up consumers, or shedding load on purpose before things break.