An engineer is designing an API endpoint that, per request, does one in-memory cache lookup, and is deciding whether to also add a synchronous call to a database in the same region versus a synchronous call to a service in a different geographic region. What rough latency numbers should they know off the top of their head to reason about this trade-off, and how do those numbers actually change the design decision?

## The numbers worth memorizing The 'latency numbers every engineer should know' - popularized by Jeff Dean's widely circulated internal Google slide and later interactive visualizations like 'Latency Numbers Every Programmer Should Know' - are a small set of orders-of-magnitude figures for how long common operations take: | Operation | Rough figure | |---|---| | an L1 cache reference | (~1ns) | | a main memory reference | (~100ns) | | an SSD random read | (tens to low hundreds of microseconds) | | a round trip within the same data center | (roughly 0.5ms) | | a round trip across a continent or ocean | (100-150ms or more) | The value of memorizing these isn't precision - actual numbers shift with hardware generations and network paths - it's having an intuitive sense of the relative gaps between them, because the gaps span roughly six orders of magnitude from memory access to intercontinental network round trip, and design decisions almost always hinge on which of these regimes an operation falls into, not its exact millisecond value. ## Using them in a design decision The mechanism for using these numbers in a design decision is straightforward but easy to skip under pressure: sum the latency of every operation on a request's critical path, weighted by how many times each happens (serially or in parallel), and identify which term dominates the total. In the scenario given: - **An in-memory cache lookup** costs on the order of 100ns - completely negligible next to anything involving a network hop, so optimizing it further is very unlikely to move the needle on end-to-end latency. - **Adding a synchronous call to a database in the same region** adds roughly 0.5ms of network round-trip time plus the database's own processing time (which itself might be another 1-10ms depending on query complexity and whether it's served from the DB's own cache or requires disk I/O) - meaningful, but usually well within budget for a typical API with a 100-300ms latency target. - **Adding a synchronous call to a service in a different geographic region** is qualitatively different: 100ms+ round trip alone can consume the entire latency budget of an interactive API, before any processing happens on the other end. ## Why the gaps exist Why these gaps exist is rooted in physics and hardware, not just software design: - **Memory access** is bound by the speed of electrical signaling across a few centimeters of circuit board and is measured in nanoseconds. - **Disk/SSD access** adds mechanical or flash-controller overhead measured in microseconds. - **Network round trips** are fundamentally bound by the speed of light over physical distance plus the overhead of every hop (routers, TCP handshakes if a fresh connection, TLS negotiation) in between. A round trip from, say, the US East Coast to Europe covers roughly 6,000-11,000 km depending on route, and light in fiber travels at about two-thirds the speed of light in a vacuum, giving a theoretical minimum round-trip time well above 40ms before any processing - real-world routing, congestion, and protocol overhead push it higher still. This is a hard physical floor: no amount of engineering cleverness removes speed-of-light latency the way caching removes disk I/O. ## The trade-off and the alternatives The trade-off this creates is central to distributed system design: keeping a call synchronous and cross-region guarantees the caller has fresh, consistent data, but it means every request pays the full network latency cost on the critical path, and that cost is irreducible. The alternatives all trade something away to avoid paying it repeatedly: - **Caching the cross-region result locally** trades data freshness for speed (the cache can be stale). - **Making the cross-region call asynchronous** (fire-and-forget, or eventual consistency via an event) trades immediate consistency for lower perceived latency. - **Replicating the data into the local region entirely** trades storage cost and replication complexity for eliminating the cross-region hop altogether. None of these are free - the right choice depends on how stale the caller can tolerate the data being and how critical low latency is to the product experience. ## The failure mode The failure mode this knowledge guards against shows up constantly in real incidents: a service adds what looks like an innocuous synchronous dependency (an auth check, a feature-flag lookup, a fraud-scoring call) to another region without accounting for the network cost, and suddenly p99 latency for every request in that path balloons by 100ms+, or - worse - if that remote region has any instability, every local request now blocks on a resource with a much larger blast radius than a local dependency would have. A well-known industry example is why content delivery networks (CDNs) and multi-region active-active deployments exist at all: companies like Netflix and major cloud providers deliberately place data and compute physically close to end users specifically to avoid paying cross-region or cross-continent round-trip latency on the critical path of every request, accepting the operational complexity of running data in multiple regions as the price of keeping that ~100ms+ term out of the user-facing latency budget.