An in-memory store answers a lookup in microseconds, but the caller opens a fresh connection for every call - what does each call pay?
answer
- setup, not service
- count the round trips first
- the handshake precedes any request
- credentials and encryption add trips
- microseconds of work, milliseconds of setup
basics
~20 sConnection setup, paid before any work begins: at least one network round trip for the handshake, plus further trips where an encrypted transport or a credential exchange is required. Microseconds of server work end up buried under milliseconds of setup.
solid answer
~50 sSplit the call into the server's service time, which is what the store spends executing, and the caller's wall time, which is what the caller measures end to end. The service time is tens of microseconds. Opening a connection first costs a handshake round trip, then whatever the store requires on top - an encrypted transport and a credential exchange each add trips - and only then is the request sent and its own round trip paid. On a same-zone tier that turns a call of roughly 0.35 ms into something nearer a millisecond, with the store doing no extra work at all. A pool amortises the setup across every later call on that connection. It does not remove the request round trip, and an idle pooled connection still occupies a slot against the store's connection ceiling.
go deeper
Recall that a call to this tier is a tiny amount of server work plus a network trip, and that connections are reused from a pool rather than opened per call. Being able to say setup is paid before any work is enough here.
Explain the setup sequence in order - handshake, any encrypted transport, any credential exchange, then the request - and do the arithmetic that shows a per-call connection costing several times a pooled one on the same tier.
Show what a pool does not fix: the request round trip per call, the standing slot each connection holds, and a liveness check on borrow that silently reinstates the cost the pool was installed to remove.
Frame connect cost as a placement and contract decision: where the tier sits relative to callers, whether an encrypted transport or credentials are required at all, and what standing connection footprint the fleet is permitted to hold.
## The two clocks in one call An in-memory store is asked for a value and answers from memory. **The server's service time** - what the store spends executing the operation - is typically tens of microseconds, because the work is a lookup in a hash table and a copy of some bytes. **The caller's wall time** - what the caller measures end to end - is a different number, and on a healthy system almost none of it belongs to the store. For a pooled connection to a tier in the same zone, the caller's wall time is roughly the sum of: - **one round-trip time** across the network, which on a same-zone hop is a few hundred microseconds; - the **serialisation** of the request and the parsing of the reply, at both ends; - the store's service time, the smallest term in the sum; - whatever the caller's own runtime adds between issuing the call and being scheduled to read the reply. Assume a same-zone round-trip time of 0.3 ms and a service time of 0.05 ms. A pooled call costs about **0.35 ms**, and the store owns roughly a seventh of it. ## What opening a connection adds Opening a connection is not one more term in that sum. It is a sequence that must run to completion **before the first byte of the request is sent**: 1. **Transport establishment** - the handshake that creates the connection costs at least one round trip, and nothing useful can be sent until it finishes. 2. **Whatever the store requires on top** - an encrypted transport adds further round trips, and a store that expects a credential adds an exchange of its own. 3. **Server-side accept work** - the store allocates per-connection state, including **the reply buffer at each end**, and consumes one slot against **the connection ceiling**, the limit the component itself sets on concurrent connections. 4. **Teardown afterwards** - closing leaves socket state on the caller's host for a while, so a high rate of connect-and-close can exhaust resources on the caller long before the store notices anything. ## The arithmetic | Cost, same-zone tier | Pooled connection | New connection per call | |---|---|---| | Transport establishment | none | about 0.3 ms, one round trip | | Encrypted transport, where used | none | about 0.3 to 0.6 ms | | Credential exchange, where required | none | about 0.3 ms | | Request round trip | about 0.3 ms | about 0.3 ms | | Server service time | about 0.05 ms | about 0.05 ms | | **Caller's wall time** | **about 0.35 ms** | **about 0.95 to 1.25 ms** | Those figures illustrate the shape; they are not any product's published numbers. Connecting per call multiplies the caller's wall time by roughly three to four **while the store does not a microsecond of extra work**. That is the entire point of the category: the cost of a design against this tier is trips, and connecting is trips. ## What a pool does not remove A pool is often described as though it made calls free. It removes one term and leaves the rest standing. - **The request round trip is still paid on every call.** A pool of a hundred connections does not make a loop of a hundred single-key reads cheaper than a hundred round trips. - **An idle pooled connection still holds a slot** against the connection ceiling and still carries per-connection buffers on the server, so a pool converts a per-call cost into a standing one. - **A liveness check on every borrow gives the saving back** - a pool that makes a call to validate a connection each time it is handed out has re-added exactly one round trip per use. Validate on idle or on a schedule instead. - **Nothing about the store's own work changes.** If a call is expensive because of what it asks for, no pool affects it. ## Where stores in this class differ The connect cost is not a single number across the class, and quoting one means quoting a product: - **Credentials vary.** Some stores of this class ship with no authentication at all and are protected only by the network they sit on; others require a credential exchange, which costs a trip at connect and nothing afterwards. - **Encryption varies.** Some tiers are reached over an encrypted transport as a matter of course, some cannot do it at all, and some sit behind an intermediary that terminates the connection, in which case the caller's connect cost is the cost of reaching the intermediary. - **The accept costs the server different amounts** depending on how large its per-connection buffers are, which is why the same connection count is comfortable on one tier and expensive on another. The claim that holds for every store in the class is the ratio: **setup is measured in round trips and the work is measured in microseconds**. Any design that pays setup per call has made the store's speed irrelevant to its own latency.
- What does a connection that is sitting idle in the caller's pool still occupy?A socket and its buffers on the caller, and on the store a slot against its connection ceiling plus the per-connection state that comes with an accepted connection, including the reply buffer at that end. Idle is cheap per call and not free in total, which is why an oversized pool is a real cost rather than a harmless margin.
- Why does validating a pooled connection on every borrow undo part of the benefit?Because a validation call is itself a round trip, added before the real request goes out. On a tier whose calls are microseconds of service time and one trip of network, that roughly doubles the caller's wall time for every call. Validate connections when they have been idle, or on a background schedule, so the check is amortised rather than per use.
Placing a phone call to say one word. The dialling, the ringing and the greeting are the cost; the word itself is free. Keeping the line open is the pool - it removes the dialling, never the speaking.
saying these in an interview costs you the question
- Calls connection setup free because the store holds everything in memory.
- Says a pool exists to save client memory rather than round trips.
- Quotes the store's service time as what the caller experiences.
- Assumes an idle pooled connection costs the store nothing at all.
- Believes every store of this class demands a credential exchange at connect.