Resilience & Stability

Pretending the exchange rate is always 1.0 can massively over- or under-charge customers without anyone noticing — it fails silently in a way that looks like success. A safer default is to block or clearly flag the transaction instead of guessing.

The health check starts too early and fails because the app isn't ready yet, so Kubernetes thinks it's broken and restarts it before it ever finishes starting up. Fix: give it a separate 'still starting' probe with a longer allowance, rather than just delaying the regular checks.

The caller no longer gets an instant answer back, it has to check later. And under real load, the answer can now take much longer to arrive because the request has to wait its turn in the queue.

Instead of one big shared limit for everybody, give each customer their own smaller limit. That way one customer sending a lot of traffic can only use up their own share, not everyone else's.

Calls that run in parallel can each get most of the budget since they finish together, but calls that run one after another must each get a smaller slice so their total doesn't exceed the time left. Splitting evenly regardless of shape wastes budget or starves later steps.

Bulkheads cost extra setup, extra resources (each pool needs its own capacity, some of which sits idle), and extra things to configure and monitor correctly. For a dependency that's low-risk, low-traffic, or extremely reliable, that overhead often isn't worth it — you'd only add a bulkhead where a failure would actually do real damage if left unisolated.

A fallback is the backup answer given when the breaker blocks a call. It should be fast, safe, and not depend on the thing that just failed. Done badly, the fallback can itself be slow, call another struggling service, or quietly hide a real problem from the people who need to know about it.

All three let you wrap a risky call so it can fail fast and recover automatically. Hystrix was the older, heavier one from Netflix, built around wrapping calls in special 'command' objects. Resilience4j is a newer, lighter Java library that wraps a plain function call instead. Polly is the .NET equivalent, using a similar lightweight wrapping style. Hystrix is no longer actively developed, so most new Java projects use Resilience4j instead.

The backup service wasn't built to handle all the traffic at once, so when everyone switched to it, it broke too. Fix it by making sure the fallback can actually handle full load, or by rate-limiting/shedding instead of just dumping everything on it.

Synthetic monitoring means scripts periodically pretend to be a user and check the app still works from the outside. If a single failed check triggers an automatic fix, like restarting servers, a flaky check or a small real problem can trigger a big automatic overreaction, like restarting everything at once, and make things worse instead of better.

The Retry-After header tells the client exactly how long to wait, so the client should honor that value instead of using its own backoff timer. If the header isn't present, the client should fall back to its normal exponential-backoff-with-jitter schedule.

If there's only one supervisor and it crashes, stuck steps stop getting fixed even though everything else works. Fix it by running more than one supervisor safely, usually with leader election so only one is active at a time.

Set the limit based on real measured capacity, with some safety margin, not a guess. Too strict and you block normal traffic for no reason; too loose and the throttle never actually kicks in before things break.

Base the timeout on the callee's real latency (like its 99th-percentile response time) plus margin, not a round number picked by guessing. If the caller waits longer than the callee itself would ever take, the caller just blocks pointlessly once the callee has already given up or hung.

Instead of asking every team to figure out bulkheads themselves, you build sane defaults into shared infrastructure (like a service mesh) that isolate every outgoing call automatically, give teams an easy way to tighten limits for their riskiest dependencies, pair it with required timeouts and circuit breakers, and monitor pool saturation everywhere so problems get caught before they cause outages.

A queue adds unpredictable waiting time, which clashes with a hard two-second promise, since the request might sit behind other work in line. For this kind of tight, synchronous deadline, it's usually better to size the service to handle real load directly, or fall back to sync-with-a-time-budget and only queue as an overflow path.

Skip it for short, simple operations that fit in one process and can just retry in place - the pattern's durable state, watchdog loop, and idempotency requirements are real infrastructure you shouldn't pay for unless the operation genuinely spans processes and needs to survive a crash mid-flight.

If every copy of your service has its own separate circuit breaker watching the same downstream dependency, they can trip and recover at different, uncoordinated times, or all probe recovery at once and re-overload the dependency. Fixes include adding randomness to timing, or moving the breaker logic to a shared layer like a service mesh so it's coordinated.

Make sure every fallback is tracked and visible, not silent; review how often they're used company-wide; and set rules for which kinds of data are allowed to use a fallback at all, so teams don't quietly guess on important things.

DNS-based failover switches which region's address gets handed out based on health checks, but DNS answers get cached everywhere, in browsers, ISPs, and apps, for minutes, so recovery is slow and uneven. If the health check itself flaps, users can end up split across two different regions at once, which is riskier than a fast, uncached, in-cluster check.

Full jitter recalculates a random delay from scratch each time based on the attempt number. Decorrelated jitter instead bases each new random delay on the previous one, which tends to spread retries out more smoothly over a long sequence of attempts.

Put limits at more than one point — at the front door, inside each service, and per customer — instead of just one place. If you only limit at one spot, problems from a different spot slip through and can still overload something downstream.

Once work is on a queue, no one is blocked waiting for it the way a synchronous caller waits on a socket, so racing the original deadline doesn't 'fail fast' for anyone. The consumer should instead use it to detect and discard work that's already too stale to matter, not to bound how long its own processing takes.