skip to content

Why is an open Kubernetes kubelet API on port 10250 a node-takeover risk, and which kubelet settings lock it down?

level: middleimportance: should knowfreq 48%

answer

  1. exec without the API server
  2. 10250 secure, 10255 read-only
  3. config file vs legacy flags
  4. TokenReview then SubjectAccessReview
  5. nodes/proxy equals remote exec

basics

~20 s

The kubelet API can run commands in any container on its node and read its logs, so anonymous access there means code execution. Disable anonymous auth, enable webhook authentication and Webhook authorization, and set readOnlyPort to 0.

solid answer

~40 s

Every kubelet runs an HTTPS API, port 10250 by default. The API server uses it for `kubectl exec`, `logs` and `port-forward`, and it can also list pods and return metrics and stats. If that API accepts anonymous requests with `AlwaysAllow`, anyone who can reach the port can run commands in every container on the node. In the `KubeletConfiguration` I set `authentication.anonymous.enabled: false`, `authentication.webhook.enabled: true`, an `x509.clientCAFile`, `authorization.mode: Webhook` and `readOnlyPort: 0`. Webhook authorization makes the kubelet ask the API server through a SubjectAccessReview, so RBAC on `nodes/proxy`, `nodes/log` and `nodes/stats` decides who gets in. One catch: these secure values are the config-file defaults, but the legacy command-line defaults are the opposite, so a kubelet started with flags only can still be wide open.

code

bash · 2 lines
bash
curl -sk -o /dev/null -w '%{http_code}\n' https://10.40.7.23:10250/pods
ss -ltn 'sport = :10255'

go deeper

for a junior

Remember that each node runs its own kubelet API on 10250, and that kubectl exec and logs go through it.

for a middle

Explain the anonymous, webhook and authorization-mode settings, the readOnlyPort, and why config-file defaults differ from the legacy flag defaults.

for a senior

Show how you would audit every node's kubelet, move collectors off 10255 without breaking dashboards, and keep nodes/proxy away from everything except the API server.

for a principal

Treat kubelet reachability as part of the node trust boundary. Decide on network segmentation, image-baked kubelet config and drift detection so that a rebuilt node cannot come back open.

## What the kubelet API is The **kubelet** is the agent on every node. Besides running pods, it serves its own **HTTPS API**, on port **10250** by default. The API server calls it for anything that must happen on the node itself: - `kubectl exec`, `kubectl attach` and `kubectl port-forward` open streams through the kubelet into a container. - `kubectl logs` reads container logs from the kubelet. - Metrics collectors read `/metrics` and `/stats` for resource usage. - `/pods` lists the pods running on the node. Older kubelets also served a **read-only port**, 10255 by default. It has **no authentication and no authorization at all**, and it exposes pod specs and metrics. ## Why an open kubelet API means node takeover If the kubelet accepts anonymous requests and authorizes them with `AlwaysAllow`, anyone who can reach port 10250 can: 1. List every pod on the node, including its environment variables and volume mounts. 2. Run a command in any container through the exec endpoint. No `kubectl`, no API server RBAC and no API audit record is involved. 3. Pick the most privileged pod on the node, such as a node agent with host mounts, and move from there to the host. So an attacker skips the API server entirely. RBAC, admission and API audit logs are all bypassed, because the request never touches the API server. ## The settings that lock it down | Setting (`KubeletConfiguration` field) | Flag | Config-file default | Legacy flag default | Hardened value | |---|---|---|---|---| | `authentication.anonymous.enabled` | `--anonymous-auth` | `false` | `true` | `false` | | `authentication.webhook.enabled` | `--authentication-token-webhook` | `true` | `false` | `true` | | `authentication.x509.clientCAFile` | `--client-ca-file` | unset | unset | the cluster CA | | `authorization.mode` | `--authorization-mode` | `Webhook` | `AlwaysAllow` | `Webhook` | | `readOnlyPort` | `--read-only-port` | `0` (disabled) | `10255` | `0` | This table explains most real incidents. **The config-file defaults are safe and the legacy command-line defaults are not.** A kubelet started from a config file with these fields left out is locked down. A kubelet started by an old script using flags only is open. kubeadm writes a config file and warns if you change these values to insecure ones. ```yaml apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration authentication: anonymous: enabled: false webhook: enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.crt authorization: mode: Webhook readOnlyPort: 0 ``` ## How Webhook authentication and authorization work - **Authentication.** A client certificate signed by `clientCAFile` identifies the caller from the certificate. The API server's own kubelet client certificate is the main example. A bearer token is sent to the API server as a **TokenReview**. The kubelet caches the result, for 2 minutes by default. - **Authorization.** The kubelet turns the HTTP request into an RBAC-style check on the **Node** resource and sends it as a **SubjectAccessReview**. The HTTP method becomes the verb, and the path becomes a subresource: - `/stats/*` becomes `nodes/stats`, `/metrics/*` becomes `nodes/metrics`, and `/logs/*` becomes `nodes/log`. - With fine-grained kubelet authorization (on by default in current releases), `/pods` checks `nodes/pods` and `/healthz` checks `nodes/healthz` first. - Everything else, including **exec, attach, run and port-forward**, becomes `nodes/proxy`. - Allowed answers are cached for 5 minutes and denied answers for 30 seconds by default, so the API server is not called on every request. The practical result: **`nodes/proxy` is a node-level remote-execution permission.** Grant it to almost no one. A monitoring agent needs `nodes/metrics` and `nodes/stats`, not `nodes/proxy`. kubeadm binds the API server's kubelet client identity to the built-in `system:kubelet-api-admin` ClusterRole, and that role is meant only for the API server. ## Checking a live node On a 27-worker cluster, check every node, not a sample. One node rebuilt from an old image is enough. - Unauthenticated `curl -k https://<node>:10250/pods` should return **401 Unauthorized**. - Nothing should be listening on 10255. - A kube-bench node run flags `--anonymous-auth`, `--authorization-mode` and `--read-only-port`. - Firewall 10250 so that only the control plane and approved collectors can reach it. This is a second layer, not a replacement for authentication. If a metrics collector breaks when 10255 closes, point it at 10250 with a ServiceAccount that has `nodes/metrics` and `nodes/stats`. Do not reopen the port.

  • Which RBAC permission should a node metrics collector get on the kubelet, and which one should it not?
    Grant `get` on `nodes/metrics` and `nodes/stats`, plus read on `nodes`. Do not grant `nodes/proxy`. The kubelet maps every path it does not recognise, including exec and port-forward, to `nodes/proxy`, so that permission lets the holder run commands in any container on any node.
  • A kubelet has anonymous auth off but authorization.mode is AlwaysAllow. Is it safe?
    No. Any caller that authenticates, for example with a ServiceAccount token that webhook authentication validates through TokenReview, can use every kubelet endpoint, including exec. Authentication only identifies the caller. Webhook authorization is what applies RBAC to that identity.

saying these in an interview costs you the question

  • Believes API server RBAC already protects kubectl exec traffic to the kubelet
  • Assumes the kubelet's read-only port still requires a token
  • Thinks kubelet defaults are the same whether set by flag or config file
  • Grants nodes/proxy to a monitoring agent that only needs metrics
  • Relies on a firewall alone and leaves kubelet anonymous auth enabled