skip to content

Your tier moves from the same zone to a second region; what happens to a path making 12 sequential calls?

level: seniorimportance: should knowfreq 50%

answer

  1. the boundary is the whole cost
  2. twelve calls, twelve crossings
  3. the tail diverges more than the mean
  4. no setting shrinks a distance

basics

~20 s

Each of the twelve crossings now pays a cross-region round trip, so a path that cost a few milliseconds costs hundreds. Distance is set by physics and topology, not configuration, so only fewer crossings or a closer tier help.

solid answer

~50 s

Placement multiplies, it does not add. Twelve sequential calls at a few hundred microseconds inside a zone is a few milliseconds; the same twelve at tens of milliseconds across regions is several hundred milliseconds, because every crossing separately pays the boundary. The tail is worse still: a long path traverses more equipment and more queues, so its high percentile diverges from its mean far more than a same-zone path does. There is nothing to tune — propagation delay is not a setting, a shorter caller-side deadline converts slow into failed rather than fast, and compression shrinks transfer time, which is the small part. The honest options are to collapse the path to one or two crossings, to give the caller's own region a tier, or to accept the latency on a path that genuinely can afford it.

go deeper

for a junior

Distance is a real cost. A call to a store in another region takes tens of milliseconds no matter how fast the store is, so a path that makes many such calls will be slow.

for a middle

Do the multiplication before and after the move and say which placement each number assumes. Notice that the store's own service time is identical in both, which tells you where the time went.

for a senior

Argue from the tail, not the mean, and show that the lever is trip count or placement rather than any setting. Be able to say what a deliberate acceptance of the latency would look like and on which paths it is defensible.

for a principal

Treat placement as a design input with a stated cost, not a deployment detail. If the second region is required, decide up front whether each region gets its own tier, and price that decision against the paths that would otherwise cross.

## What moves when the boundary moves The cost of one crossing is dominated by distance and by how much equipment sits in between. Orders of magnitude, which you should confirm against your own path: - **Same host, over a local socket** — tens of microseconds. There is no network. - **Same zone** — a few hundred microseconds. One or two hops of switching. - **Different zone in the same region** — around a millisecond. A short physical distance plus more equipment. - **Different region** — tens of milliseconds, sometimes far more. Set by the distance between the two places and the route between them. The last row is the one that breaks designs, because it is two orders of magnitude above the first and **no software setting affects it**. A signal takes the time it takes. ## Twelve calls, twelve crossings | Placement | One round trip | 12 sequential calls | Server time inside all 12 | |---|---|---|---| | Same zone | ~0.3 ms | ~4 ms | well under 1 ms | | Different zone | ~1 ms | ~12 ms | well under 1 ms | | Different region | ~40 ms | ~480 ms | well under 1 ms | The last column does not move. The store is doing exactly the same work it always did; every millisecond that appeared came from the boundary, multiplied by the number of times the path chose to cross it. This is why placement and trip count have to be reasoned about together: twelve crossings is unremarkable in one row and a broken page in another. ## The tail, not the mean A long path is not just a slower path — it is a more variable one. - It traverses more devices, and every device is a queue that can be busy. - It is more likely to be affected by a congested or re-routed segment. - A retransmit costs a full extra crossing, and on this path a crossing is enormous. So the high percentile of a cross-region crossing sits much further above its mean than a same-zone crossing's does, and a path that repeats the crossing twelve times is likelier to include at least one bad one. A budget written from the mean will be met on average and missed exactly when someone is watching. ## Why there is no tuning lever Teams reach for four things here, and none of them works: 1. **Client and connection settings.** They change how a crossing is made, not how long the distance is. 2. **A shorter caller-side deadline on one operation.** This converts a slow request into a failed one. It protects the caller's own budget, which is sometimes the right call, but it does not deliver the value any sooner. 3. **Compression.** It reduces transfer time, which is a small component of a long crossing; the propagation delay is untouched. 4. **A faster or larger store.** The microseconds it saves are invisible next to the milliseconds the boundary costs. ## The three honest options - **Collapse the path.** Twelve crossings becoming one — one operation taking many keys, or a run sent without waiting for each reply — takes about 480 ms back to about 40 ms. The distance is still paid, but once. - **Put a tier in the caller's own region.** This removes the boundary from the request path entirely. It then raises a separate question that this arithmetic does not answer: what each region's tier holds, and whether anything keeps copies in step. - **Accept it deliberately.** A background job that runs once an hour can afford 480 ms. A page render cannot. The point is that it becomes a stated choice with a number attached, not an accident. ## What varies, and what to check before you trust the number - **Some paths already cross a boundary nobody intended.** A managed tier may sit in a different zone from the caller by default, and an extra proxy or gateway in front of it is another crossing with its own cost. - **Some stores acknowledge a write only once another copy holds it**, and where that copy is decides whether a write pays the boundary even when the reader does not. Others acknowledge locally, so reads and writes cost the same. Ask which before pricing a write path. - **A partitioned tier can make the count worse**, because a path that made twelve crossings to one node may make more to several. - **Measure, do not assume.** The round-trip figures above are shapes. The only number worth designing against is one measured from the caller's host at a high percentile.

  • The team proposes raising the caller's deadline on one operation from 50 ms to 500 ms so the path stops failing. What does that achieve?
    It converts a failing path into a very slow one. The value still arrives no sooner, the caller now holds its resources for ten times as long, and any deadline further up the stack may fire first anyway. A longer deadline is the right tool when a slow answer is genuinely more useful than none; it is never a fix for a path that crosses a region twelve times.
  • Which part of the path should you measure to prove the boundary is the cost?
    Two numbers taken at the caller: the wall time of a single trivial call at the new placement, and the number of calls the path makes. Multiply them and compare against the observed request time. If the product accounts for most of it, the boundary and the trip count are the whole story, and the store's own service time will confirm it by being unchanged.

saying these in an interview costs you the question

  • Expects a client or connection setting to recover cross-region round trips
  • Prices the move using the average crossing rather than its tail
  • Assumes twelve calls stay cheap because each is a single read
  • Treats a region boundary as just another network hop
  • Claims a faster serialiser or compression will absorb the distance