Why does a Kubernetes EndpointSlice still list a deleted pod's IP, and is that pod still getting new Service traffic?

## What an EndpointSlice actually records A **Service** with a label selector does not route traffic by itself. The **EndpointSlice controller**, which runs inside `kube-controller-manager`, watches the pods that match the Service's selector and writes their addresses into one or more **EndpointSlice** objects. Each slice carries the label `kubernetes.io/service-name` naming its Service, so you can list them with a label selector. Every entry in a slice's `endpoints` list has one or more `addresses`, a `targetRef` pointing at the pod, a `nodeName`, and a `conditions` block. That block is the part that matters here, because **being listed and being eligible for new traffic are two different things**. ## The three conditions and how the controller sets them For a pod-backed endpoint, the EndpointSlice controller computes three booleans: | Condition | Set from | Meaning for consumers | |---|---|---| | `serving` | the pod's `Ready` condition | the application says it can take traffic | | `terminating` | whether the pod has a `metadata.deletionTimestamp` | the pod is on its way out | | `ready` | `serving` and not `terminating` | the endpoint should receive new traffic | One exception changes `ready`: if the Service sets `spec.publishNotReadyAddresses: true`, the controller marks every endpoint `ready: true` regardless of readiness or termination. That is meant for peer discovery in clustered systems, not for normal client traffic. The API also defines how to read a missing value: a nil `ready` or `serving` means **true**, and a nil `terminating` means **false**. The EndpointSlice controller always sets all three explicitly for pods. ## Why a deleted pod is still listed When you run `kubectl delete pod`, the API server does not remove the pod object at once. It sets `deletionTimestamp` and the pod enters its grace period. The pod still has an IP and is not yet in a terminal phase, so the controller **keeps it in the slice** and flips its conditions: - `terminating` becomes `true` straight away; - `ready` becomes `false` straight away, because a terminating endpoint is never ready (outside `publishNotReadyAddresses`); - `serving` keeps tracking the pod's `Ready` condition, which usually stays `true` until the application stops passing its readiness probe. The same logic covers a pod that fails readiness without being deleted: `serving` and `ready` both become `false`, `terminating` stays `false`, and the pod remains listed. That combination means "alive but not taking traffic", which consumers can tell apart from "shutting down". The entry disappears only when the pod object is finally gone or reaches a terminal phase. Keeping it visible is deliberate: it lets consumers tell "draining" apart from "vanished" and make a better choice when nothing else is left. ## Who still sends it traffic Consumers such as kube-proxy route **new** connections only to `ready` endpoints. So in the normal case a terminating pod stops receiving new Service traffic **once each consumer has processed the update**; that propagation is not instant. During a rollout of the accrual service on a 16-node cluster, some nodes can still hold rules that include the old pod for a short while after the slice changes. Two further details: 1. **Fallback.** If a Service has no `ready` endpoints at all, kube-proxy falls back to endpoints that are both `serving` and `terminating`, rather than dropping every connection. For `externalTrafficPolicy: Local` it applies the same fallback per node. 2. **Existing connections.** Removing an endpoint from the ready set affects new connections. A TCP connection that was already established keeps flowing to the same pod until one side closes it. ## How to check The default table from `kubectl get endpointslices` prints an `ENDPOINTS` column with **every** address in the slice, terminating ones included, so it cannot answer the question. `kubectl describe endpointslice` prints `Ready` for each endpoint but not `serving` or `terminating`. The reliable view is the object itself: ```bash kubectl get endpointslices -n loyalty -l kubernetes.io/service-name=points-accrual -o yaml kubectl get endpointslices -n loyalty -l kubernetes.io/service-name=points-accrual -w ``` In the YAML, look for the entry whose `targetRef.name` is the deleted pod and read its `conditions`. During a rollout of the loyalty-points accrual Deployment, `-w` shows the slice being rewritten as each old pod flips to `terminating: true` and a replacement becomes `ready: true`. ## Common misreadings - Treating the `ENDPOINTS` column as the routing table. - Assuming a deleted pod is removed from the slice in the same instant, instead of being marked. - Forgetting that `publishNotReadyAddresses` makes `ready` true even for terminating pods. - Reading the legacy **Endpoints** object and expecting the same condition detail; it only splits addresses into ready and not-ready lists.