skip to content

After wrapping each net.Conn in a session recorder, an SSH bastion's throughput collapsed. What did the wrapper break?

level: seniorimportance: must knowfreq 52%

answer

  1. the arriving value has a new type
  2. embedding satisfies the interface, not the capability
  3. an optional probe now returns false
  4. the kernel path needs the concrete connection
  5. measure throughput with and without the wrapper

basics

~20 s

The wrapper's dynamic type is no longer the concrete connection, so probes for optional interfaces such as io.ReaderFrom now fail and the kernel-assisted copy path is gone. Every byte moves through a user-space buffer instead. Nothing reports an error.

solid answer

~50 s

Embedding `net.Conn` in a recorder makes the wrapper satisfy `net.Conn`, which is why it compiles and why nothing complains. But capability probes look at the dynamic type you are actually holding, and that is now the recorder. `io.Copy` between the two sides of the bastion used to find `*net.TCPConn`'s `ReadFrom` and let the kernel splice bytes from one socket to the other; against the wrapper the probe fails, so every byte is read into a 32 KiB buffer and written back out, with two extra copies and two syscalls per chunk. The same disappearance hits other capabilities the concrete type had — `CloseWrite() error` for half-closing a session, and `SyscallConn` — and losing half-close is a hang, not a slowdown. Confirm it by measuring end-to-end throughput on a large transfer with and without the wrapper. Fix it by forwarding the capability, or by wrapping only the direction you actually record.

code

go · 11 lines
go
type tap struct {
	net.Conn
	log io.Writer // nil when this direction is not recorded
}

func (t *tap) ReadFrom(r io.Reader) (int64, error) {
	if t.log != nil {
		r = io.TeeReader(r, t.log)
	}
	return io.Copy(t.Conn, r) // probes the inner conn, not the wrapper
}

go deeper

for a junior

Recall that a wrapper type has only the methods it declares or embeds, so anything extra the wrapped value could do is no longer visible from outside.

for a middle

Explain how a failed capability probe changes which code path runs, and why that change produces no error anywhere — only different performance.

for a senior

Demonstrate the diagnosis: measure throughput and CPU per byte with and without the wrapper, identify the lost capability, and choose between forwarding, narrowing the wrap, and exposing the inner value.

for a principal

Own the policy question: what your platform requires wrappers to forward, how that requirement is reviewed and tested, and where recording is worth its cost at all.

## What the wrapper did The recorder looks harmless: ```go type recordConn struct { net.Conn log io.Writer } func (c *recordConn) Read(b []byte) (int, error) { n, err := c.Conn.Read(b) c.log.Write(b[:n]) return n, err } ``` Embedding promotes every `net.Conn` method, so `*recordConn` satisfies `net.Conn` and every call site keeps compiling. What changed is invisible in the type system: the value now flowing through the program has the method set of `*recordConn`, which is `net.Conn`'s methods and nothing more. Every method the *concrete* connection had beyond that — and every optional interface built on those methods — is gone from the outside world's view. ## Why throughput collapses A bastion is two connections and a copy in each direction, typically `io.Copy(dst, src)`. `io.Copy` probes: source for `io.WriterTo`, then destination for `io.ReaderFrom`. Unwrapped, one of those probes hits the concrete TCP connection, which on Linux implements `ReadFrom` in terms of the kernel's splice/sendfile machinery: bytes move between two file descriptors without ever entering your address space. That is one syscall per large chunk, no user-space buffer, and no allocation. With the wrapper, both probes fail. `io.Copy` allocates a 32 KiB buffer and runs the generic loop: `read` syscall into the buffer, copy of the bytes into your process, `write` syscall out. For a bastion pushing bulk data, that is a large multiple of the syscalls and a fresh copy of every byte, and it shows up as CPU time in the proxy process and as a throughput ceiling for the session. ## Why nobody noticed at build time This is the defining property of optional capability probing: **the failure is silent**. There is no compile error — the wrapper satisfies the declared interface. There is no run-time error — a failed probe is a normal outcome and simply selects the fallback. There is no vet diagnostic, because forwarding an optional interface is a design obligation, not a language rule. The only signal is performance, and only if someone measures. ## Diagnosing it The honest diagnostic here is an end-to-end throughput measurement, taken twice: push a large, known payload through the bastion with the recorder enabled and with it disabled, on the same hosts and the same path, and compare bytes per second and CPU seconds per gigabyte. A wrapper that removed a kernel path shows a step change in both, and the step is proportional to bytes moved, which distinguishes it from a per-connection or per-request regression. Confirm the mechanism rather than guessing: a CPU profile of the proxy will show time in the copy loop and in memory moves that were not there before, and a syscall count per megabyte will have jumped. ## The correctness half, which is worse than the speed half Speed is the visible symptom; the dangerous loss is behavioural. `*net.TCPConn` has `CloseWrite() error`, and forwarding proxies probe for it — usually with an inline `interface{ CloseWrite() error }` — to half-close one direction when the other side sends EOF. Wrapped, the probe fails, the proxy falls back to a full `Close` or to nothing at all, and sessions either truncate or hang until a timeout. The same applies to `SyscallConn`, used by code that needs the raw descriptor. Losing an optimisation costs money; losing half-close costs correctness. ## Fixes, in order of preference 1. **Wrap narrowly.** Record only the direction you must record. The other direction keeps its raw connection and its kernel path. Most session recording only needs one side captured in full. 2. **Forward the capability.** Give the wrapper the methods it hides, delegating to the embedded value: ```go func (c *recordConn) CloseWrite() error { if cw, ok := c.Conn.(interface{ CloseWrite() error }); ok { return cw.CloseWrite() } return c.Conn.Close() } ``` Note the shape: the wrapper probes what it holds, so a wrapper around a connection that never had the capability still behaves sanely. 3. **Expose what you hold.** Add an `Unwrap()` accessor and document it, so a caller that needs the underlying value can walk to it. This is the only fix that survives capabilities invented after your wrapper was written, since you cannot forward a method you have never heard of. ## The tension you must state out loud You cannot both observe every byte and let the kernel move them without you. For the direction you genuinely record, the fast path is unrecoverable by definition, and the right answer is to say so and to scope the recording as tightly as the requirement allows. The mistake in the incident was not recording; it was wrapping *both* directions, and every connection, when the requirement covered one.

  • How would you prove the wrapper is the cause rather than the network or the peer?
    Run the same large transfer over the same path with the recorder compiled in and compiled out, and compare bytes per second and CPU seconds per gigabyte end to end. A capability loss scales with bytes moved and burns CPU inside the proxy, whereas a network problem shows up as latency or loss without the CPU change, and a peer problem does not follow the build.
  • Besides io.ReaderFrom, what else does wrapping a TCP connection hide?
    `CloseWrite() error`, which proxies probe for to half-close one direction — losing it turns a clean shutdown into a hang. Also `SyscallConn`, used to reach the raw descriptor, and any concrete tuning methods on the connection type such as read-buffer and linger settings. All of them vanish the moment the outermost value is your wrapper.
  • Can a wrapper simply forward every optional interface it might hide?
    No. It can only forward capabilities its author knew about, so a capability added later is hidden again, and a wrapper that must observe the bytes cannot forward a path that bypasses it. That is why the durable fix is an accessor exposing the wrapped value, with forwarding used for the specific capabilities that matter today.
  • If the recorder must capture both directions, what is left to optimise?
    Not the kernel path — observing the bytes rules it out for the recorded direction. What remains is reducing the work per byte: reuse one buffer across the session rather than allocating per copy, size it for the workload, write the recording asynchronously so the log's latency does not sit in the data path, and drop the wrapper entirely for sessions that policy does not require you to record.

Putting the parcel in a plain outer box: it still fits every shelf, but the express lane no longer recognises the label that got it moved without being unpacked.

saying these in an interview costs you the question

  • Says the wrapper cannot matter because it still satisfies net.Conn
  • Blames the network without measuring with and without the wrapper
  • Expects a compile error or a panic when a capability is hidden
  • Thinks a bigger copy buffer restores the kernel path
  • Overlooks the lost half-close and treats it as a speed issue only
  • Wraps every connection in both directions when only one is recorded