A service registered in Consul's service mesh has a sidecar proxy running, but its outbound calls still go straight to the destination's real address and never get mTLS. How is an application normally expected to address an upstream in Consul's mesh, and what does transparent proxy mode change?
answer
- the mesh does not capture traffic by default
- one local bind port per declared upstream
- the app dials 127.0.0.1, not the real address
- transparent mode redirects with iptables
- virtual.consul addresses for un-declared destinations
basics
~20 sBy default Consul does not intercept outbound traffic. Each upstream is declared in the sidecar registration with a local_bind_port, and the application must dial 127.0.0.1 on that port. Transparent proxy mode instead installs iptables rules so all outbound traffic is redirected into the sidecar.
solid answer
~50 sConsul's default mesh model is **explicit upstreams**. In the service registration you declare a `connect { sidecar_service { proxy { upstreams = [...] } } }` block, each entry pairing a `destination_name` with a `local_bind_port`; the sidecar opens a listener on `127.0.0.1:<port>`, and the application is expected to dial *that*, not the real service. The sidecar then does discovery, load balancing and mTLS on its behalf. If the app instead resolves and dials the destination's real address, nothing intercepts it — there is no iptables redirection in this mode — so the traffic leaves the box unencrypted, is not authorized by intentions, and looks to you like a mesh that is simply not working. **Transparent proxy** mode changes that: Consul installs iptables rules that redirect the workload's outbound (and inbound) traffic into the sidecar, so the application can keep using ordinary service addresses and still get mesh treatment. It is enabled with `Mode = "transparent"` on `proxy-defaults` or `service-defaults`, and it is the default on Kubernetes.
code
hcl · 21 linesservice {
name = "web"
port = 8080
connect {
sidecar_service {
proxy {
upstreams = [
{
destination_name = "db"
local_bind_port = 9191
},
{
destination_name = "payments"
local_bind_port = 9192
}
]
}
}
}
}go deeper
Know that with a Consul sidecar the application usually connects to a port on localhost, and that the proxy is what actually reaches the real service on the other side.
Explain the upstreams block with destination_name and local_bind_port, why a call to the real address silently skips the mesh, and what transparent proxy mode does instead.
Diagnose the un-meshed call from evidence: absent inbound proxy metrics, plaintext on the wire, intentions apparently ignored. Know how redirection is installed and how to verify it inside the workload's namespace.
Decide the mesh onboarding model across a mixed VM and Kubernetes estate: explicit upstreams for auditability versus transparent redirection for adoption speed, and what egress policy applies to traffic leaving the mesh.
## The default: the mesh is opt-in per destination Unlike meshes that capture everything a pod sends, Consul's original model asks the application to cooperate. When you register a service with a sidecar, you list the destinations it will call: ```hcl service { name = "web" port = 8080 connect { sidecar_service { proxy { upstreams = [ { destination_name = "db" local_bind_port = 9191 } ] } } } } ``` The sidecar — started by `consul connect envoy -sidecar-for=web`, which fetches a bootstrap configuration and execs the proxy — opens a listener on `127.0.0.1:9191`. The application's database URL must therefore point at `127.0.0.1:9191`, not at `db.example.internal:5432`. Everything else happens behind that local socket: the proxy resolves healthy instances of `db`, picks one, opens a mutual-TLS connection to that instance's sidecar, and the destination sidecar checks intentions before handing the connection to its own application. ## Why the symptom in the question happens If the application keeps its original connection string, the outbound packet never touches the proxy. There is no interception in this mode, so: - the connection is plaintext, not mTLS; - no intention is consulted, because intentions are enforced by the *destination's* inbound sidecar and this connection is not arriving there in mesh form; - there are no proxy metrics for the call, so dashboards make it look as if the dependency is barely used. The fix is either to repoint the application at its local bind port, or to switch the service to transparent proxy mode. A related trap: `local_bind_port` is per upstream, so adding a new dependency means adding a new upstream entry and a new port — configuration that has to be kept in step with the application's own config. That coupling is exactly what transparent proxy removes. ## Transparent proxy mode With `Mode = "transparent"` set on a `proxy-defaults` (mesh-wide) or `service-defaults` (per service) config entry, Consul stops relying on the application's cooperation: ```hcl Kind = "proxy-defaults" Name = "global" TransparentProxy {} Mode = "transparent" ``` An init step — on Kubernetes, an init container running Consul's traffic-redirection setup — installs iptables rules in the workload's network namespace that redirect outbound traffic to the sidecar's outbound listener and inbound traffic to its inbound listener. Now the application can dial a destination by its normal name and still get discovery, mTLS and intention enforcement. To make that work for names it has never seen, Consul assigns each mesh service a **virtual IP** and answers for `<service>.virtual.consul`, and on Kubernetes it can map the ordinary Service address onto the mesh destination. Transparent proxy is the default when Consul is installed on Kubernetes; on VMs, explicit upstreams remain common because there is no pod network namespace to program per workload. ## What each model costs you Explicit upstreams are predictable and require no privileged network setup: you can see, from the registration alone, every destination a service is allowed to dial. They are also tedious, and they invite exactly the drift described above, where the application's configuration and the sidecar's upstream list disagree. Transparent proxy is far less work per service but hides the plumbing. When it misbehaves — traffic to an external host being swallowed, or a port not redirected — you are now debugging iptables rules inside a network namespace rather than reading a config file. It also raises the question of what should happen to traffic bound *outside* the mesh; Consul can be told to allow only mesh destinations (a `mesh` config entry setting `TransparentProxy { MeshDestinationsOnly = true }`) so that an accidental call to the public internet fails loudly instead of silently leaving. ## What to check when mesh traffic is missing Confirm the sidecar is registered and running; confirm which mode the service is in; in explicit mode check that the application's target really is the local bind port; in transparent mode check that the redirection rules were installed at all (a workload started without the init step will look completely un-meshed). In both cases the destination sidecar's inbound metrics are the honest signal: if they show no connections while the application clearly is talking to the dependency, the traffic is going around the mesh.
- With explicit upstreams, what happens if two services declare the same `local_bind_port` for different destinations?Nothing conflicts between services, because the bind is on each workload's own loopback interface — the port only has to be unique within one workload's network namespace. Inside a single service, however, two upstreams cannot share a port: the second listener fails to bind and that dependency becomes unreachable through the proxy.
- Under transparent proxy, how does the sidecar know which mesh service an outbound packet was meant for?Consul assigns each mesh service a virtual IP and resolves it under `<service>.virtual.consul`, and on Kubernetes it maps the Service address onto the mesh destination. The redirected connection therefore carries a destination address the proxy can map back to a service name, which it then resolves through the discovery chain.
- Should traffic to a destination outside the mesh be allowed to pass through transparent proxy untouched?That is a policy choice. Consul can restrict redirected traffic to mesh destinations only, via the `mesh` config entry's `TransparentProxy { MeshDestinationsOnly = true }`. Turning it on makes accidental egress fail loudly instead of leaving silently, at the cost of having to model every legitimate external destination — typically behind a terminating gateway.
saying these in an interview costs you the question
- Assumes Consul intercepts all outbound traffic by default
- Thinks declaring an upstream rewrites the app's DNS resolution
- Says the sidecar encrypts traffic the app sent to the real address
- Confuses the upstream's local bind port with the destination's real port
- Believes transparent proxy needs no privileged setup in the namespace