skip to content

How do an in-process instrumentation agent, kernel-level observation and a sidecar proxy differ in what telemetry each can capture?

level: seniorimportance: nice to knowfreq 26%

answer

  1. Three vantage points, three blind spots
  2. Inside, beneath, and beside the process
  3. Only one sits among your own objects
  4. Encryption hides payloads from outside observers
  5. A proxy sees only what it routes

basics

~20 s

Three vantage points, three blind spots. An in-process agent sees application objects and can carry trace context; kernel-level observation sees sockets and syscalls in any language but no application meaning; a sidecar proxy sees only traffic that crosses the network.

solid answer

~50 s

An **in-process agent** stands among your objects: it can read call arguments, follow work across thread pools, and copy trace context from an inbound request onto the outbound calls it causes - but it must be built per runtime and it needs a restart. **Kernel-level observation** attaches beneath the process to sockets, syscalls and scheduling events, so it works for any language and any binary without a restart, but it sees bytes rather than meaning and payloads are opaque once TLS is involved. A **sidecar proxy** parses protocols on the network path, giving every service in the estate an identical vocabulary for method, path, status and duration - but it is blind to everything inside the process, and it cannot link the request it received to the request it later sent unless the application forwards the trace-context headers itself. Most estates run more than one and give each a job.

code

text · 5 lines
text
inbound  (proxy span)  POST /orders          trace T1   <- from caller's headers
   app code calls payments, does NOT forward the trace-context headers
outbound (proxy span)  POST /charge          trace T2   <- proxy started a new one

result: two correct spans, two separate traces, no parent-child link

go deeper

for a junior

Recall that telemetry can arrive without editing code in more than one way: something loaded inside the process, something watching from the operating system, or a proxy on the network path. Being able to name the three is enough here.

for a middle

Explain the mechanics of each: where it attaches, whether the process must restart, and whether it is tied to one language runtime. Be able to say what a network-path observer fundamentally cannot see inside a process.

for a senior

Demonstrate the propagation asymmetry - only the in-process vantage point naturally links an inbound request to the outbound calls it caused, which is why proxy-only tracing yields disconnected per-hop spans when applications drop context.

for a principal

Own the estate-level combination: which mechanism is your unavoidable floor, where you buy depth, what privileged node-level access costs politically, and how much per-request overhead an extra network hop is worth at peak traffic.

## Three places you can stand All three of these produce telemetry without anybody editing application code, which is why they get compared. They differ in where they stand relative to the process, and that single fact determines what each one can and cannot know. **Inside the process.** An instrumentation agent loaded into the runtime wraps recognised library call sites. It sits among your application's own objects, so it can read call arguments, follow work across thread pools and callbacks, and — crucially — write trace-context headers onto outbound requests and read them off inbound ones. **Beneath the process.** Kernel-level observation attaches to the operating system rather than to the application: socket reads and writes, connection lifecycle, process and scheduling events, syscall entry and exit. It is indifferent to the language the process is written in and does not require the process to restart. **Beside the process.** A sidecar or gateway proxy sits on the network path and parses the protocols flowing through it. Every request in and out of the workload passes through one implementation, which parses method, path, status, peer and duration identically for every service in the estate. ## What each one can see | | In-process agent | Kernel-level | Sidecar proxy | | --- | --- | --- | --- | | Language coupling | One build per runtime | None | None | | Needs a restart | Yes | No | Yes, to inject the sidecar | | Sees in-process time | Yes, at hooked calls | Partly, as on/off-CPU time | No | | Sees application meaning | Yes, if a plugin exists | No | Only what the protocol carries | | Sees encrypted payloads | Yes, before encryption | Not without user-space hooks | Yes, if it terminates the connection | | Can continue a trace itself | Yes | No | Only per hop | | Deployment unit | Per service | Per node, privileged | Per workload | ## The propagation asymmetry — the point interviewers push on Producing a span is easy from all three vantage points. Producing a *connected* trace is not, and this is where the three diverge sharply. Only the in-process agent sits where the incoming request and the outgoing request it caused are both visible as the same unit of work, so only it can copy the trace context from one to the other without help. Kernel-level observation sees two socket conversations and has no way to know that one caused the other. A proxy sees an inbound request and, later, an outbound request from the same workload — and it must guess, or it must rely on the application forwarding the incoming trace-context headers on its own outbound calls. That is why mesh-only tracing so often produces neat pairs of disconnected spans: the proxy did its job perfectly on each hop, and the application dropped the context in between. **A proxy gives you per-hop facts for free; it does not give you a trace for free.** ## Where each one is blind - **In-process agent** — anything with no plugin, and every language you have not built an agent for. A Rust batch job and a vendor binary get nothing. - **Kernel-level** — application semantics. It can tell you a socket carried 4,096 bytes in 63 ms; it cannot tell you that was a menu lookup for a pupil in year seven. Payload contents are opaque once TLS is in play unless it also hooks the crypto library in user space, which reintroduces per-library coupling. - **Sidecar proxy** — everything that never crosses the network. A 612 ms in-process rules evaluation is invisible, and so is any traffic routed around the proxy. ## Costs, honestly stated The in-process agent costs engineering per runtime, start-up time, and a real interference risk inside your process. Kernel-level observation costs privileged access on every node, coupling to what each node's kernel supports, and a per-node rollout that is a platform team's problem rather than a service team's. A sidecar costs CPU and memory beside every workload plus an extra network hop on both directions of every call — at 5,400 requests per second that hop is a capacity line item, not a rounding error. ## How they combine in practice Most mature estates run more than one and assign each a job: 1. The proxy provides the uniform, unavoidable baseline — every hop, every service, one vocabulary, including services nobody has touched in two years. 2. The in-process agent provides depth and connected traces for the runtimes where an agent exists and the services that matter. 3. Kernel-level observation covers what neither reaches: third-party binaries, unfamiliar languages, and connection-level facts during an incident where redeploying is not on the table. Choosing is therefore less "which one" and more "what is my floor, and where do I buy depth above it". A candidate who names only one mechanism has usually only worked in one language estate.

  • Which of the three can tell you that 612 ms of a request was spent inside your own code?
    Only the in-process one, and only where it has hooks or you added spans yourself. Kernel-level observation can show that the process was on CPU rather than blocked, which is a real clue but not an attribution to a logical operation. A proxy sees only the wall time between its own two hooks and cannot decompose it at all.
  • When would you deploy kernel-level observation even though an in-process agent is available?
    When the target is unreachable by an agent: a third-party or vendor binary you cannot rebuild, a language with no agent, or a fleet where restarting ninety services to attach something is not an option. It is also the pragmatic choice when you want one uniform floor of connection-level facts across a heterogeneous estate.
  • Why does a sidecar proxy produce more consistent span naming than agents do?
    Because one implementation handles every hop. The names come from a single protocol parser rather than from a different plugin in each language, so the vocabulary is uniform by construction. The catch is that the uniformity covers only the fields visible on the wire - nothing about which internal path the request took.

Three witnesses to a delivery: a colleague standing in the room, a technician reading the door sensors, and a guard at the gate. Each is honest, and none of them saw the same thing.

saying these in an interview costs you the question

  • Thinks a sidecar proxy can see in-process function timings
  • Assumes kernel-level observation reads encrypted payloads by default
  • Believes a service mesh removes the need to forward trace context
  • Treats in-process agent coverage as language-independent
  • Says kernel-level collection needs no privileged access
  • Picks one mechanism as universally correct for every workload