An Istio VirtualService http route can carry a `mirror` destination alongside its normal route. What exactly does the proxy send to the mirror target and what does it do with the response, and why is mirroring live production traffic risky?

seniorshould knowfreq 42%

answer

a copy, and nobody waits for it
the answer goes in the bin
the authority header is marked
the copy still commits its writes
doubling load on shared dependencies

basics

~20 s

The proxy sends a fire-and-forget copy of the request to the mirror target with -shadow appended to the Host header, and discards the response entirely, so the client is unaffected. The risk is that the mirrored copy is a real request: it repeats writes and side effects on whatever the mirror target touches.

solid answer

~50 s

Mirroring (traffic shadowing) duplicates a matched request to a second destination while the original is routed normally. The copy is fire and forget: the proxy does not wait for it, and the mirror's response — success, error or timeout — is discarded and never reaches the client. To make the copy identifiable, the proxy appends -shadow to the authority/Host header, so payments becomes payments-shadow. mirrorPercentage controls what share of matched requests is copied, which matters because mirroring at 100% doubles load on the target and on everything it calls. The real danger is that the mirrored request is not a simulation: if the shadow deployment shares a database, a queue or a payment provider with production, mirrored writes commit, messages are published and vendors are called for real. Safe mirroring means an isolated data path, a target that recognises the -shadow authority, and a percentage low enough that the shared dependencies survive.

Mirroring (traffic shadowing) duplicates a matched request to a second destination while the original is routed normally. The copy is **fire and forget**: the proxy does not wait for it, and the mirror's response — success, error or timeout — is discarded and never reaches the client. To make the copy identifiable, the proxy appends **`-shadow`** to the authority/Host header, so `payments` becomes `payments-shadow`. `mirrorPercentage` controls what share of matched requests is copied, which matters because mirroring at 100% doubles load on the target and on everything it calls. The real danger is that the mirrored request is not a simulation: if the shadow deployment shares a database, a queue or a payment provider with production, mirrored writes commit, messages are published and vendors are called for real. Safe mirroring means an isolated data path, a target that recognises the `-shadow` authority, and a percentage low enough that the shared dependencies survive.

code

yaml · 20 lines

yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payments
spec:
  hosts:
    - payments
  http:
    - match:
        - method:
            exact: GET
      route:
        - destination: {host: payments, subset: v1}
      mirror:
        host: payments
        subset: v2
      mirrorPercentage:
        value: 5.0
    - route:
        - destination: {host: payments, subset: v1}

go deeper

for a junior

Know that a mirrored request is a fire-and-forget copy whose response is discarded, so the user's request and its latency are unaffected.

for a middle

Explain the mechanics precisely — the -shadow suffix on the authority header, mirrorPercentage for sampling — and say why the copy is a real request rather than a replay.

for a senior

Lead with the failure modes: mirrored writes committing, the shadow's own outbound calls being real, doubled load on shared dependencies, and real user data reaching a less-hardened environment. Then give the controls in order.

for a principal

Decide when shadowing is worth its cost against synthetic load or a small canary, and set the rules that keep it safe: isolated data paths, shadow detection in the service, sampling defaults, and an expiry so no mirror outlives its purpose.

What the feature does

A mirror is attached to an ordinary HTTP route entry in a VirtualService:

yaml

    - route:
        - destination: {host: payments, subset: v1}
      mirror:
        host: payments
        subset: v2
      mirrorPercentage:
        value: 5.0

The primary route behaves exactly as it would without the mirror. Additionally, the proxy makes a copy of the matched request and sends it to the mirror destination. Three properties define the behaviour:

Asynchronous. The proxy does not wait for the mirror. Client latency is unchanged in the normal case.
Response discarded. Whatever the mirror returns — 200, 500, or nothing at all — is thrown away. The mirror cannot fail the user's request, and it cannot succeed on its behalf either.
Marked. The copy's authority/Host header has -shadow appended, so the receiving side can tell it apart from real traffic.

mirrorPercentage takes a percentage value, so you can shadow a sample rather than everything.

Why it is genuinely useful

Mirroring is how you get production-shaped input into a new version without exposing users to it. Real request bodies, real header combinations, real distribution of sizes and endpoints — the things synthetic load never reproduces. You watch the shadow's error rate, latency and logs, and if it falls over, no user notices. It is also the cheapest way to test a rewrite of a service against the traffic that actually exists rather than the traffic you assumed.

Why it is risky

The copy is a real request, not a replay into a sandbox. Everything the mirror target does, it does for real:

Writes. If the shadow deployment points at the production database, mirrored POSTs insert rows, mirrored PUTs overwrite, mirrored DELETEs delete. Duplicate charges and duplicate emails are the folklore examples for a reason.
Downstream calls. A shadow's own outbound calls are ordinary mesh traffic. It will call the same vendors, publish to the same topics and warm — or evict from — the same caches.
Load. Mirroring at 100% doubles the request rate on the target and on every shared dependency underneath it. If the shadow shares a database with production, you have doubled the database's load in the name of a safe test.
Data exposure. Mirrored requests carry real user data, including credentials in headers, into an environment that may be newer, less hardened and more verbosely logged.
Backpressure. The copy is fire-and-forget from the client's point of view but not free for the proxy: it still consumes connections and pool capacity toward the mirror target, so a slow or overwhelmed shadow can consume connection-pool slots you cared about.

Making it safe

The controls follow directly from the risks:

Isolate the data path. The shadow should read from and write to its own database, its own queues and stubbed third parties. This is the one non-negotiable.
Use the -shadow marker. Because the authority header is marked, the target can detect shadow traffic and short-circuit anything with an external side effect. This is the defence in depth for when isolation is imperfect.
Start low. Begin at a small mirrorPercentage and raise it while watching the shared dependencies, not just the shadow.
Scope the match. Mirror a read-heavy subset of endpoints first. There is rarely a reason to shadow the payment path on day one.
Remember it is running. A forgotten mirror is a permanent, invisible multiplier on some service's load, and it will be discovered during an unrelated incident.

Where it sits among the neighbouring tools

Mirroring is the observe-only end of a spectrum the VirtualService also covers at the other end: fault injects deliberate delay or abort into a percentage of matched requests, letting you verify that callers actually handle a slow or failing dependency. Both are percentage-scoped and both should be matched narrowly — commonly on sourceLabels or a test header — so that only traffic you intend is affected. Mirroring answers "can the new version handle what production sends it?"; fault injection answers "can the callers handle it when this dependency misbehaves?"

One version note worth having: recent Istio releases add a mirrors list supporting several mirror destinations with their own percentages, alongside the original single mirror plus mirrorPercentage fields. Check which form your version's API accepts before writing the manifest.

## What the feature does A mirror is attached to an ordinary HTTP route entry in a VirtualService: ```yaml - route: - destination: {host: payments, subset: v1} mirror: host: payments subset: v2 mirrorPercentage: value: 5.0 ``` The primary route behaves exactly as it would without the mirror. Additionally, the proxy makes a copy of the matched request and sends it to the mirror destination. Three properties define the behaviour: 1. **Asynchronous.** The proxy does not wait for the mirror. Client latency is unchanged in the normal case. 2. **Response discarded.** Whatever the mirror returns — 200, 500, or nothing at all — is thrown away. The mirror cannot fail the user's request, and it cannot succeed on its behalf either. 3. **Marked.** The copy's authority/Host header has `-shadow` appended, so the receiving side can tell it apart from real traffic. `mirrorPercentage` takes a percentage value, so you can shadow a sample rather than everything. ## Why it is genuinely useful Mirroring is how you get production-shaped input into a new version without exposing users to it. Real request bodies, real header combinations, real distribution of sizes and endpoints — the things synthetic load never reproduces. You watch the shadow's error rate, latency and logs, and if it falls over, no user notices. It is also the cheapest way to test a rewrite of a service against the traffic that actually exists rather than the traffic you assumed. ## Why it is risky The copy is a **real request**, not a replay into a sandbox. Everything the mirror target does, it does for real: - **Writes.** If the shadow deployment points at the production database, mirrored POSTs insert rows, mirrored PUTs overwrite, mirrored DELETEs delete. Duplicate charges and duplicate emails are the folklore examples for a reason. - **Downstream calls.** A shadow's own outbound calls are ordinary mesh traffic. It will call the same vendors, publish to the same topics and warm — or evict from — the same caches. - **Load.** Mirroring at 100% doubles the request rate on the target *and* on every shared dependency underneath it. If the shadow shares a database with production, you have doubled the database's load in the name of a safe test. - **Data exposure.** Mirrored requests carry real user data, including credentials in headers, into an environment that may be newer, less hardened and more verbosely logged. - **Backpressure.** The copy is fire-and-forget from the client's point of view but not free for the proxy: it still consumes connections and pool capacity toward the mirror target, so a slow or overwhelmed shadow can consume connection-pool slots you cared about. ## Making it safe The controls follow directly from the risks: - **Isolate the data path.** The shadow should read from and write to its own database, its own queues and stubbed third parties. This is the one non-negotiable. - **Use the `-shadow` marker.** Because the authority header is marked, the target can detect shadow traffic and short-circuit anything with an external side effect. This is the defence in depth for when isolation is imperfect. - **Start low.** Begin at a small `mirrorPercentage` and raise it while watching the shared dependencies, not just the shadow. - **Scope the match.** Mirror a read-heavy subset of endpoints first. There is rarely a reason to shadow the payment path on day one. - **Remember it is running.** A forgotten mirror is a permanent, invisible multiplier on some service's load, and it will be discovered during an unrelated incident. ## Where it sits among the neighbouring tools Mirroring is the observe-only end of a spectrum the VirtualService also covers at the other end: `fault` injects deliberate `delay` or `abort` into a percentage of matched requests, letting you verify that callers actually handle a slow or failing dependency. Both are percentage-scoped and both should be matched narrowly — commonly on `sourceLabels` or a test header — so that only traffic you intend is affected. Mirroring answers "can the new version handle what production sends it?"; fault injection answers "can the callers handle it when this dependency misbehaves?" One version note worth having: recent Istio releases add a `mirrors` list supporting several mirror destinations with their own percentages, alongside the original single `mirror` plus `mirrorPercentage` fields. Check which form your version's API accepts before writing the manifest.

How can the mirror target itself avoid performing side effects it should not?
By checking the authority/Host header, which the proxy marks with a `-shadow` suffix on mirrored copies. The service can short-circuit writes, outbound vendor calls and message publishing when it sees that suffix. It is defence in depth — the primary control is still an isolated database and stubbed third parties.
What is the argument for setting mirrorPercentage well below 100 even when the shadow is fully isolated?
Isolation of the shadow's own datastore does not isolate everything it touches — shared caches, shared vendors, cluster capacity and the proxy's connection pools all absorb the copy. Mirroring at 100% doubles that load. Starting small lets you watch the shared dependencies rather than only the shadow's own error rate.
Istio's VirtualService also offers fault injection. How does that differ in intent from mirroring?
Mirroring is observe-only — a copy goes elsewhere and the user's request is untouched. Fault injection deliberately degrades the real request, adding a `delay` or returning an `abort` status to a percentage of matched traffic, to prove the callers handle it. One tests the new version against real input; the other tests callers against real failure.
A mirror was configured months ago and forgotten. What does that cost?
A permanent invisible multiplier on some service's load and on every dependency it shares, usually discovered during an unrelated incident when capacity numbers do not add up. Mirrors deserve an expiry the way a feature flag does: a named owner, a reason, and a date by which the route entry is removed.

It is a wiretap that also dials the number: you hear everything the line carries, but the person on the other end is genuinely called each time.

saying these in an interview costs you the question

Thinks the mirror's response is compared with the primary's
Believes mirrored requests cannot write to a database
Assumes mirroring adds no load to shared dependencies
Mirrors all traffic at 100 percent as a first step
Says the client waits for the mirror, adding latency

An Istio VirtualService http route can carry a `mirror` destination alongside its normal route. What exactly does the proxy send to the mirror target and what does it do with the response, and why is mirroring live production traffic risky?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

What the feature does

Why it is genuinely useful

Why it is risky

Making it safe

Where it sits among the neighbouring tools

▾common follow-ups4 questions

▾analogy~10 s

▾red flags5 flags

§ context

▾related questions4 questions

▾related topics4 topics

What the feature does

Why it is genuinely useful

Why it is risky

Making it safe

Where it sits among the neighbouring tools

answer

go deeper

context