When injecting a fault into a service's call to a downstream dependency, what is the practical difference between injecting added latency and injecting error responses, and why does latency injection usually uncover more bugs?

## Two faults that feel similar and behave nothing alike Both experiments target the same dependency, but they exercise different machinery. **Error injection** makes the dependency answer wrongly and quickly: an HTTP 500 or 503, a connection refused, a gRPC `UNAVAILABLE`. The call completes almost instantly. The resources it held are released. What is under test is the code path *after* the failure: is there a fallback, is it correct, does the retry policy fire, does the failure get counted against the right metric, does the user see something sane? **Latency injection** makes the dependency answer *correctly but late*: a fixed or distributed delay before the normal response. Nothing errors. Every call simply stays open longer, and while it is open it holds a thread, a connection from the pool, a buffer, and a slot in whatever queue sits in front of it. ## Why slow beats broken, quantitatively Little's law is the tool for this: `concurrency = throughput x latency`. A service handling 200 requests per second against a dependency answering in 100 ms needs 200 x 0.1 = **20** concurrent slots. Inject 2 seconds of delay and the same throughput needs 200 x 2 = **400** slots. If the client pool is sized at 50, the other 350 requests queue, then time out, then — if a retry policy is naive — come back and make it worse. The critical consequence is that saturation is **shared**. A fast error is contained inside the one request that touched the failing dependency. A slow response consumes a resource that other, unrelated requests need. If the whole application shares one servlet thread pool or one HTTP connection pool, an outage in a non-critical recommendations call can stall checkout. That is the failure this experiment is designed to expose, and it is exactly the argument for per-dependency isolation of pools. ## Choosing the delay value: aim just past the timeout A random delay teaches you little. Pick values relative to the caller's own configuration: - **Just under the timeout** (say 90% of it) — proves the timeout is not accidentally firing on normal-ish slowness, and shows what the latency budget upstream looks like when a dependency is merely unhealthy. - **Just over the timeout** — the highest-value single value. It forces the timeout to fire and reveals whether the timeout path, the retry policy, and the fallback all behave. - **Far past the timeout, or indefinite** — this is where you discover calls with **no timeout configured**, which is the single most common finding of a latency experiment. If the injected delay is 60 seconds and requests hang for 60 seconds, some layer has no bound. Many client libraries default to an unbounded read timeout, so this is not a hypothetical. Also vary the *shape*: a fixed delay on 100% of calls is a clean signal, but a delay applied to a small percentage of calls models a partially degraded dependency — one bad replica behind a load balancer — and that reveals whether your alerting and your outlier ejection notice a tail problem at all, which an all-or-nothing outage would mask. ## What error injection is still uniquely good for Latency does not replace error injection; it answers a different question. Errors are the right fault when you want to test: - **Which errors are retried.** Retrying a 500 may be correct; retrying a 400 is a bug; retrying a non-idempotent write after a timeout can double-charge a customer. - **Correct status semantics.** Does a downstream 503 surface as a 503 or does it leak as a 500 and pollute your own error budget? - **Fallback correctness.** A fallback value that is silently wrong is worse than an error, and only an error path exercises it. - **Error-rate alerting.** Injecting a 5% error rate is the honest way to check whether your alerting actually fires at the threshold you claim. ## Blackhole: the third option people forget Between "fast error" and "slow success" sits the nastiest real-world fault: packets dropped with no response at all. A `REJECT` produces an immediate connection-refused — that is error injection at the network layer. A `DROP` produces a blackhole: the caller's TCP connect or read hangs until *its own* timeout, or until the OS TCP retransmission limit, which can be minutes. Blackholing is the most faithful model of a hung dependency, a lost network path, or an overloaded box that accepted the connection and never got to it, and it is what you inject when you want to know whether connection timeouts and read timeouts are both set. ## The order to run them in If you can only run one experiment against a dependency, run latency. It subsumes much of what error injection tests (once the timeout fires you are on the error path anyway) *and* it reaches the resource-saturation behaviour that errors never touch. Run explicit error injection afterwards to check the semantics of specific status codes and the correctness of fallbacks.