Messaging & Integration

All the workers need to speak the same language for the data passing between them — same fields, same format. If one worker expects data shaped differently than the last worker sends it, they can't be swapped or reused.

Instead of always doing important work first no matter what, you can either give a small guaranteed share of time to the less-important queue (like 1 in 10 turns), or make old low-priority messages count as more important the longer they wait, so they eventually get processed.

In cloud migrations, the ACL is often its own small deployed service (or gateway) that talks to the old on-prem system and the new cloud services, translating between them while the old system is gradually replaced. Originally the same idea was described as code inside a team's own codebase for keeping domain models from mixing.

The translator can quietly mis-map data when the outside system changes without telling you, the extra layer can slow things down or become a single point of failure, and it can turn into a dumping ground for random logic if nobody owns it. Guard against these with tests, monitoring, and clear ownership.

Polling means the client keeps checking back for updates; a callback (webhook) means the server calls the client back when the work is done. Polling is simpler to set up on the client side; callbacks are faster and less wasteful but need the client to run something reachable.

When many workers pull from one line, a later job can finish before an earlier one if it lands on a faster worker. To keep related jobs in order, you route them all to the same one worker instead of letting any worker grab them.

One bad message that always fails can get retried forever and clog things up. The fix is to count how many times it's failed, and after a limit, move it out of the main queue into a separate 'failed' queue so a human can look at it later, instead of retrying forever.

If all five calls share the same limited pool of workers, the slow service can hog enough workers that the gateway can't process calls to the other four healthy services either, so one bad dependency drags down everything. Fix it by giving each downstream its own separate pool or limit and a circuit breaker, so one slow service can only exhaust its own slice.

The gateway can save a copy of a response and reuse it for later identical requests, saving backend work, but only if the response is truly the same for everyone who asks; if the response depends on who's asking (like personal data), caching it for all users is a serious bug.

Routing rules can overlap and steal each other's traffic, point at a service that's gone, or fall out of sync across gateway instances. To on-call, this looks like plain errors with no obvious cause, since the backends are healthy.

With several workers grabbing messages at once, a worker that gets an easy message can finish before another worker still stuck on a harder one — so messages can come out in a different order than they went in. That's fine for some jobs, but breaks ones where order matters, like applying account changes in sequence.

If just going first isn't enough anymore, you can also refuse or delay some of the low-priority work entirely — like temporarily rejecting or throttling non-urgent requests — so the important stuff always has enough room. It means some low-priority work gets sacrificed on purpose.

If a retried step or compensation isn't idempotent, running it twice can double-charge, double-refund, or double-cancel something. Saga steps need unique operation IDs so duplicates can be detected and safely ignored.

Skip it when the outside system's model is already close to yours, the integration is small, short-lived, or has one consumer, or you're about to fully replace the outside system anyway. Retire an existing ACL once nothing is left using the system it protects against.

For work that's usually fast, forcing every client through an accept-then-poll dance adds unnecessary round trips and complexity for the common case. Better options: keep it synchronous with a generous timeout for the fast path, or only switch to async for inputs that are actually large or slow.

You have to decide how fast to add or remove workers as the pile of waiting jobs grows or shrinks. Add workers too slowly and jobs pile up; add them too fast and you might overload the systems those workers depend on, or pay for capacity you don't need.

Putting all these shared jobs in one gateway makes every service simpler, but now that one gateway becomes critical: if it's slow, wrong, or down, everything behind it is affected, and one team owns something every other team depends on.

If the job is small, simple, needs to be lightning-fast, or a small team already struggles to run one service — splitting it into many pieces connected by queues adds cost and complexity that isn't worth it. Sometimes one well-written function does the job better.

If the different kinds of work are similar enough in urgency, or the system is small, adding separate priority queues can just be extra complexity for no real benefit. Sometimes it's simpler and cheaper to just give everything enough capacity, or to run truly urgent work on its own completely separate system instead.

You can build one aggregator per client type, each owned by that client's team, one shared aggregator for everyone, or replace hand-written aggregation with a query language like GraphQL where the client asks for exactly the fields it needs. Each trades ownership clarity and flexibility against operational overhead.

One hand-edited routing table doesn't scale once many teams own routes and traffic spans regions. Routes need per-team ownership, dynamic publishing as services change, and sometimes a mesh of proxies per service instead of one central gateway.

A semantic lock is a flag added to a record (like 'pending') while a saga is still in progress, so other parts of the system know not to fully trust or freely modify that record until the saga finishes.