skip to content

Requests to a hostname served by a Kubernetes ingress controller come back with 404 from the controller itself rather than from the application. Walk through how you would diagnose it.

level: seniorimportance: should knowfreq 44%

answer

  1. classify: controller 404 vs app 404
  2. curl -H 'Host: ...' straight at the LB IP
  3. host in spec.tls ≠ routing rule
  4. 404 = rules, 502/503 = backends
  5. longest prefix wins across merged Ingresses

basics

~20 s

A 404 from the controller means no rule matched. Check that the Ingress has the right class and was claimed, that the Host header matches spec.rules[].host exactly, that the path and pathType actually cover the request, and that the backend Service and its EndpointSlices resolve. Confirm the request reached the expected controller.

solid answer

~60 s

A controller-generated 404 (often a default-backend page) means routing never selected a backend. Work outside in. 1. **Did the request reach the right controller?** Resolve the hostname and compare with the controller Service's external address. Curl the controller directly with an explicit Host header to bypass DNS: `curl -H 'Host: shop.example.com' http://<lb-ip>/path`. 2. **Was the Ingress claimed?** `kubectl get ingress` — an empty ADDRESS or a class the running controller does not serve means it was never programmed. 3. **Host mismatch.** `spec.rules[].host` matches the Host/SNI value exactly; a wildcard `*.example.com` does not cover the apex, and a request to the IP sends no matching Host at all. 4. **Path mismatch.** `Prefix` matches whole segments, so `/foo` never matches `/foobar`; `Exact` is case-sensitive and trailing-slash-strict. Remember the longest prefix and any Exact rule win over your intended rule. 5. **Backend resolution.** `kubectl get endpointslices` for the Service — no ready endpoints usually shows as 503, a wrong port as a connection error, so a 404 more often points at rules than at pods. Controllers' logs and default-backend metrics confirm which rule (if any) matched.

code

bash · 12 lines
bash
dig +short shop.example.com
kubectl get svc -n ingress-nginx ingress-nginx-controller -o wide

curl -sS -o /dev/null -w '%{http_code}\n' \
  -H 'Host: shop.example.com' http://<controller-ip>/api/orders

curl -sS --resolve shop.example.com:443:<controller-ip> \
  -o /dev/null -w '%{http_code}\n' https://shop.example.com/api/orders

kubectl get ingress -A -o wide | grep example.com
kubectl describe ingress shop -n shop
kubectl get endpointslices -n shop -l kubernetes.io/service-name=api

go deeper

for a junior

Check that the Ingress exists with the right host and path, that the Service name and port are correct, and that pods are running.

for a middle

Add class/claim verification, exact host matching including wildcard rules, and the Prefix segment semantics that make a plausible-looking rule not match.

for a senior

Lead with classifying who returned the 404, bypass DNS with an explicit Host header, reason about precedence across merged Ingress objects, and use the 404-versus-502/503 signature to separate rule problems from backend problems.

for a principal

Turn it into prevention: per-host ownership so objects cannot silently conflict, controller-level metrics for default-backend hits, synthetic checks per hostname, and admission policy that rejects duplicate host/path claims.

## First, classify the 404 The single most useful step is deciding **who produced it**. A 404 from your application means routing worked and the app disagreed about the path; a 404 from the controller means no rule matched and the request fell to the default backend. Distinguishing them is usually easy: controllers return a recognisable default page or a `Server:` header naming the proxy, and application 404s carry the app's own body and headers. Check the controller access log for the request — if it appears with the default backend as upstream, it is the controller's. That classification instantly halves the search space. Everything below assumes a controller 404. ## Step 1 — Did the request arrive at the controller you think? DNS is not managed by the Ingress. Resolve the hostname and compare with the address on the controller's Service (or the address in the Ingress status). In clusters with an internal and a public controller, a very common cause is DNS pointing at the *other* one, which has no rule for that host and therefore 404s correctly. Bypass DNS to test: `curl -H 'Host: shop.example.com' http://<controller-address>/path`, and for HTTPS use `curl --resolve shop.example.com:443:<ip>` so SNI is right. If the correct controller answers properly, the problem is DNS or the load balancer, not the rules. ## Step 2 — Was the object programmed at all? `kubectl get ingress -A` and check ADDRESS. Empty means no controller claimed it. Then confirm `spec.ingressClassName` names an IngressClass whose `spec.controller` matches a running controller — a typo or an unset class with no default class produces silent non-service. `kubectl describe ingress` shows a `Sync` event from a controller that owns the object; total silence is diagnostic. If the class is right, verify the controller actually loaded it: NGINX-family controllers let you dump the generated configuration from the pod, and most expose the parsed rule set in logs at higher verbosity. ## Step 3 — Host matching `spec.rules[].host` is compared to the Host header (and to SNI for TLS). Points of failure: - Requesting by IP sends no useful Host, so only a rule with **no** host set can match. - Wildcards are single-label and DNS-style: `*.example.com` matches `shop.example.com`, but not `example.com` and not `a.b.example.com`. - The host in `spec.tls[]` is about certificate selection, not routing — listing a host there does not create a rule for it. A host present only in `tls` and not in `rules` gives exactly this symptom: TLS completes, then 404. - Trailing dots, uppercase, or a port suffix in the Host header can matter depending on controller normalisation. ## Step 4 — Path matching Re-read the rule with the semantics in mind, not the intent: - `Prefix` is **segment-wise**. `/foo` matches `/foo`, `/foo/`, `/foo/bar`, but never `/foobar`. - `Exact` is case-sensitive and does not tolerate a trailing slash difference. An `Exact: /app` rule 404s `/app/`. - Precedence is `Exact` first, then the **longest** matching prefix — independent of YAML order and merged across multiple Ingress objects for the same host. Another team's newer, longer prefix can capture a subtree you expected, so list every Ingress for the host, not just yours: `kubectl get ingress -A -o wide | grep example.com`. - `ImplementationSpecific` paths may be regexes; a regex that fails to anchor as expected is a frequent cause. - Rewrite annotations can transform the path *after* matching; if the rewritten path is wrong you get an **application** 404, which is the other side of the classification in step 1. ## Step 5 — Backend and endpoints Check the Service named in the backend exists in the **same namespace as the Ingress** (Ingress cannot reference a Service in another namespace) and that its port number or name matches the Service definition, not the container port. Then `kubectl get endpointslices -l kubernetes.io/service-name=<svc>` to confirm ready addresses. Note the signature difference: missing or unready endpoints normally yields **503**, a wrong target port yields connection errors or 502, and a missing Service can make the controller drop the rule entirely — which loops back to a 404. So a 404 mostly indicts rules, while 502/503 indict backends. Stating that distinction is what separates a rehearsed answer from an experienced one. ## Step 6 — The multi-object view Finally, look at the whole host across the cluster. Two Ingresses for the same host in different namespaces, a stale object left behind by a deleted release, or a duplicate served by a second controller can all mean the rule you are editing is not the rule serving traffic. Some controllers refuse to merge conflicting definitions and keep the older object, so a newly applied Ingress silently does nothing — visible only in controller logs or an `ingress conflict` style event. ## Fast checklist Classify the 404 → confirm which controller answered → confirm the object was claimed → host exact match → path semantics and precedence → endpoints → competing objects for the same host.

  • How would you tell a 404 produced by the ingress controller from one produced by the application?
    Compare the response body and headers against the controller's known default backend page and its `Server` header, and look for the request in the controller's access log with the default backend recorded as upstream. If the controller logged the request against your real Service, routing succeeded and the application returned the 404 — which usually means a rewrite produced a path the app does not serve.
  • Everything looks correct in the Ingress, yet traffic is unaffected by your edits. What else is going on?
    Something else is serving that host: another Ingress object for the same hostname in a different namespace, a leftover object from an old release, or a second ingress controller that DNS actually points at. Controllers merge rules per host and some keep the oldest definition on conflict, so your object can be accepted yet inert. List every Ingress for the hostname cluster-wide and check which controller the address belongs to.

saying these in an interview costs you the question

  • Jumping straight to pod logs without establishing whether the controller or the app returned the 404
  • Assuming a host listed in spec.tls is automatically routable
  • Testing only through DNS, so a wrong controller or stale record is never ruled out
  • Reading Prefix matching as a string startsWith and concluding the rule should have matched
  • Expecting a 404 when the real cause is missing endpoints, which normally surfaces as 503

context