How would you decide which share of a service's HTTP tests runs in-process versus against a real port?
answer
- cheapest layer that can see the fault
- select by fault class, not by route
- each wire test names the property it defends
- slow suites stop being run
- topology change means revisit the list
basics
~20 sPut the bulk in-process, where cases cost fractions of a millisecond, and reserve wire tests for fault classes only a connection can reveal - framing, TLS, timing, cancellation, proxy trust - one case per class rather than per route.
solid answer
~50 sStart from what each style can prove. In-process tests are cheap and deterministic and cover everything above the dispatcher, so routes, binding, chain order, statuses and bodies belong there, exhaustively. Wire tests cost startup, a real client and a share of flakiness, and they are the only way to see framing, transport security, timing, backpressure, cancellation and whatever the deployment puts in front of the process. So select them by **fault class**, not by endpoint: each wire test should name the property it defends, and a second test defending the same property is a duplicate. Watch both failure modes - a suite with no wire coverage meets the listener configuration for the first time in production, and one that drifts toward the wire gets slow enough that people stop running it. Revisit the split when the deployment topology changes, because that is what the wire set encodes.
go deeper
Remember the default shape: most HTTP cases run in-process because they are nearly free, and a small number go over a real port for what only a connection can show.
Justify placement per fault class - which defects are visible above the dispatcher and which need bytes, a clock or a peer - rather than quoting a ratio.
Talk about the operating costs you have paid: startup time, port and shutdown flakiness, harder failure triage, and the maintenance a topology change forces on the wire set.
Own the policy: name the fault classes, place each at the cheapest layer, state what only a deployed environment can prove, and describe the pressure that stops the expensive set from crowding out fast feedback.
## Frame it as buying coverage of fault classes The wrong question is "how many integration tests should we have". The useful one is: **which fault classes can hurt us, and what is the cheapest test that can see each one?** That reframing does the selection for you, because the two styles have a clean boundary. | Fault class | Cheapest test that can see it | |---|---| | Route matching, method fallback, parameter binding | In-process | | Body deserialization and its failure path | In-process | | Hook chain order, short-circuits, error mapping | In-process | | Status, headers and body of a response | In-process | | Request parsing, size limits, header fidelity | Wire | | Body and response framing | Wire | | Transport security and how the application reacts to it | Wire | | Progressive delivery, backpressure, disconnect cancellation | Wire | | Listener-enforced timeouts | Wire | | Trust of what a hop in front adds | Wire | Everything in the top half should be covered **exhaustively** in-process, because each case costs almost nothing. Everything in the bottom half needs **one deliberate case per property**, not one per route. ## The cost model that drives the ratio 1. **Runtime.** An in-process case costs a fraction of a millisecond; a wire case costs a server start plus milliseconds to seconds each. A thousand of the first is cheaper than twenty of the second. 2. **Flakiness.** Ports, startup races, client timeouts and shutdown ordering all add failure modes the in-process path does not have. Flaky tests get muted, and muted tests protect nothing. 3. **Debuggability.** An in-process failure gives one stack trace on the test thread. A wire failure gives a status or a client error with the cause on a server worker, costing minutes of investigation each time. 4. **Rot.** Wire tests encode deployment shape, so a topology change invalidates them. That is a feature - they are where the shape is asserted - but it is maintenance the fast suite does not carry. ## The two failure modes to steer between - **No wire coverage at all.** Every assumption about the listener, framing and whatever sits in front is first tested in production. The fast suite is green by construction, because it asserts the assumptions back at you. - **Drift toward the wire.** "Integration tests are more real" is seductive, and the set grows one endpoint at a time until the suite takes long enough that developers stop running it locally and CI feedback arrives too late to act on. A slow suite is a weaker suite, whatever it covers. The steering rule that survives both: **every wire test must name the wire property it defends.** If it cannot, it duplicates something the fast suite already covers and should be deleted or moved down. ## Governance that keeps the split honest - Keep the wire set **listed and small enough to read in one screen**, with the property each case defends written next to it. - Treat a topology change - a terminating hop added, security moved, a protocol version changed - as a trigger to revisit the whole list, since those tests exist to encode exactly that. - Run the fast suite on every change and the wire set on every change to the listener or deployment configuration, plus on a schedule. - Add an **outer ring** beyond both: a post-deploy check against the real environment. Some properties - the real terminator, real network conditions, the real client population - cannot be reproduced locally at any price, and pretending otherwise inflates the wire set without closing the gap. - When a production incident is traced to a fault class no test could have seen, add **one** case at the cheapest layer that can see it, and say which class it now defends. ## What a good answer sounds like Not a ratio. A principal-level answer names the fault classes the service can suffer, places each at the cheapest layer that can catch it, admits which ones only production can prove, and describes the pressure that keeps the expensive set from growing - because the long-run risk is not too few integration tests, it is a suite so slow that the fast feedback everyone actually relied on stops being run.
- What is the argument against simply running everything over the wire?Runtime, flakiness and debugging cost all rise while coverage of framework-level behaviour does not improve at all - those faults were already caught above the dispatcher. The practical result is a suite slow enough that people stop running it locally, which removes the fast feedback that caught most defects.
- How do you keep the wire set from growing one endpoint at a time?Require each case to name the connection-level property it defends, and reject one that names a property another case already covers. Selection by fault class caps the set at roughly the number of such properties, while selection by endpoint has no natural ceiling.
- Which properties can neither style prove locally?Anything that depends on the real environment: the actual terminating hop and its configuration, real network conditions and client populations, and the deployed listener's settings. Those need a post-deploy check against the environment itself, which is why the wire set should not be stretched to imitate them.
saying these in an interview costs you the question
- Argues integration tests are always more valuable
- Picks a fixed ratio with no reference to fault classes
- Adds a wire test per endpoint rather than per property
- Ignores that a slow suite gets muted or skipped
- Assumes local wire tests can stand in for the real environment