Resilience & Stability

A circuit breaker watches calls to another service. Normally it's 'closed' and lets calls through. If too many fail, it 'opens' and stops calling that service for a while, failing fast instead. After a timeout it lets a few test calls through ('half-open') to check if the service recovered.

Skip or replace the broken part instead of crashing everything — show checkout without recommendations, or with generic ones, so the customer can still buy. This is called graceful degradation, and the substitute response is a fallback.

A liveness probe checks if an app is stuck and needs restarting. A readiness probe checks if it's ready to handle traffic right now. Failing liveness means restart the container; failing readiness means stop sending it requests, but leave it running.

A queue lets the web app drop off work instantly and move on. Workers pull jobs from the queue at a steady pace they can handle, so a flood piles up safely instead of crashing the service.

Retrying instantly and repeatedly can flood a struggling service with even more traffic, making things worse. Exponential backoff means waiting longer between each retry (like 1s, 2s, 4s, 8s) so the service gets breathing room to recover.

The scheduler starts and tracks a multi-step job, agents do the actual remote work for each step, and the supervisor watches for stuck or failed steps and decides what to do about them.

Throttling means a service limits how much work it accepts. If too many requests come in at once, it slows down, delays, or rejects some so it doesn't crash and can keep serving everyone reasonably well.

A connect timeout limits how long you wait to open the network connection; a read timeout limits how long you wait for the response after the connection is open. Different failure modes need different limits.

A thread-pool bulkhead runs each call on its own small set of dedicated threads, so a stuck call can't block the caller's own thread. A semaphore bulkhead just counts how many calls are in flight and blocks new ones over the limit, but they still run on the caller's own thread.

The breaker looks at the last batch of calls. The failure-rate threshold trips it if too many of those calls errored out. The slow-call threshold trips it separately if too many calls succeeded but took too long. Both are percentages over a recent batch, not a single call.

Half-open lets only a small number of test calls through, not all traffic. If enough of those test calls succeed, the breaker fully reopens for business (closes). If too many fail, it goes back to blocking everything (reopens). Limiting the number of test calls avoids overwhelming a dependency that might still be fragile.

Serving old cached data keeps the site up but customers might see wrong prices or 'in stock' items that sold out. Fix it by capping how old the cached data can be and clearly marking it as possibly outdated.

A shallow check just says 'the app process is up.' A deep check also verifies things like the database connection work. Deep checks catch more real problems, but can make a shared outage, like the database going down, take down every instance at once if used carelessly.

Each of the 10,000 requests becomes a message sitting in the queue almost instantly. Workers then pull and process 200 of those messages per second, so the queue drains over about 50 seconds instead of the workers being hit with 10,000 requests at once.

The first request might have actually succeeded even though the client never got a response, so retrying it blindly could charge the customer twice. Making the operation idempotent — using a unique request ID so the server can recognize and ignore a duplicate — prevents that.

If every client waits the same amount of time before retrying, they all retry at once, creating a new spike. Jitter adds randomness to each client's wait time so retries spread out instead of arriving together.

Because a timeout doesn't guarantee the original call failed - it might have already succeeded. Retrying a non-idempotent operation can make it run twice, like charging a customer or sending an email a second time.

The supervisor watches a shared record of each step's status and deadline. If the deadline passes with no 'done' status, or the agent explicitly reports an error, the supervisor treats it as stuck or failed.

A hard limit is a strict cap that's never crossed, even if there's spare capacity. A soft limit is a flexible cap that can be exceeded temporarily if the system has room, but tightens up when things get busy.

Deadline propagation passes the actual 'give up by this time' moment from the original caller down through every hop, so downstream services know exactly how much time is left, instead of each hop guessing its own fixed timeout independently.

All backends share the same pool of 100 connections. Since search accepts connections but never replies, more and more of those 100 connections get stuck talking to search and never come back to the pool. Eventually there are none left for the other eleven backends, so everything fails, not just search. Giving search its own separate slice of connections would keep the rest working.

Keep the things people actually came for (continue watching, account access) working the longest, and turn off the fancier extras (personalized recommendations, trending rows) first since the page still works fine without them.

The load balancer's health check and Kubernetes' internal readiness check run on separate schedules and separate systems, so there's a lag between Kubernetes deciding a pod is gone and the load balancer noticing. If a pod shuts down before the load balancer catches up, some requests get sent to a pod that no longer exists, causing failed requests during deploys or scale-downs.

They only handled short bursts, not a sustained flood: orders kept arriving faster than workers could process them for two whole hours, so the backlog never had a chance to shrink. They should have watched queue depth and message age and had a way to add more workers or slow producers when the backlog kept climbing instead of finding out only after storage filled up.

If every layer retries 3 times independently, one failure at the bottom can multiply into dozens of retries flowing back up through the chain — a retry storm. A retry budget caps the fraction of a service's total traffic that's allowed to be retries, so it stops amplifying once that cap is hit.

It can retry, hand the step to a different worker or resource, escalate to a human, or - if earlier steps already had real effects - undo them with a compensating action. Which one it picks depends on whether the failure looks temporary, whether a different resource could succeed, and whether anything needs undoing.

You can say no right away, make the request wait a bit, or give a simpler/cheaper answer instead of the full one. Saying no is cheap but annoys the caller; waiting is nicer but risky if too many wait at once; a simpler answer keeps things running but might be less accurate.

Not automatically. The client giving up just means it stops waiting; the server keeps working unless something explicitly tells it to stop, like a cancellation signal sent over the still-open connection, or the server checking a shared deadline or context object during its own work and bailing out early.

You size it from how fast the dependency normally responds and how many calls you expect at once — enough slots to handle normal peak traffic, but not so many that one slow dependency could still eat all your resources. Too small rejects healthy traffic; too large defeats the isolation.