How does a Kubernetes kubelet get and rotate its certificates through CertificateSigningRequests, and why do rotateCertificates and serverTLSBootstrap differ at approval?

## Two certificates, two directions Every kubelet needs two different certificates: - A **client certificate** to authenticate *to* the API server. Its subject is `CN=system:node:<nodeName>`, `O=system:nodes`, which the Node authorizer and RBAC recognise as that node. - A **serving certificate** for its own HTTPS endpoint (port 10250), which the API server calls for `kubectl logs`, `exec`, `port-forward` and metrics. Both can be issued through the **CertificateSigningRequest (CSR)** API (`certificates.k8s.io/v1`). A CSR object holds a PKCS#10 request, a `spec.signerName`, the usages and an optional `spec.expirationSeconds` (minimum 600). It moves through `Approved` or `Denied` conditions before a signer writes `status.certificate`. ## TLS bootstrap of the client certificate A new worker in the 9-node bare-metal cluster behind the IoT ingest gateway starts with no client certificate at all. The flow is: 1. The kubelet starts with `--bootstrap-kubeconfig`, which holds a short-lived bootstrap credential. The token itself belongs to cluster bootstrap. 2. It creates a CSR with signer `kubernetes.io/kube-apiserver-client-kubelet`. 3. The `csrapproving` controller in kube-controller-manager recognises a kubelet client request. It runs a SubjectAccessReview that checks whether the requester may `create` the `certificatesigningrequests/nodeclient` subresource (for a first certificate) or `selfnodeclient` (to renew its own). kubeadm binds the ClusterRoles `system:certificates.k8s.io:certificatesigningrequests:nodeclient` and `...:selfnodeclient` to grant those. 4. The `csrsigning` controller signs with the CA given by `--cluster-signing-cert-file`/`--cluster-signing-key-file` (the cluster CA in kubeadm). The certificate lifetime is capped by `--cluster-signing-duration`, which defaults to 8760h. 5. The kubelet writes `/var/lib/kubelet/pki/kubelet-client-current.pem` and uses it from then on. ## Rotation The `rotateCertificates` field of the KubeletConfiguration defaults to `false` in the kubelet API, but kubeadm always sets it to `true`. With rotation on, the kubelet's certificate manager picks a deadline at a **jittered 70-90% of the certificate's lifetime**. At that point it generates a new key, submits a new CSR, and switches to the new certificate once it is signed. The jitter keeps a batch of nodes that joined together from all renewing in the same minute. Because renewals authenticate with the still-valid current certificate, the `selfnodeclient` check passes and approval is automatic. The trap: rotation works only while the old certificate is still valid **and** the control plane is reachable. A node powered off for longer than the remaining lifetime comes back with an expired credential and must be bootstrapped again. ## The serving certificate is different | | Client cert | Serving cert | |---|---|---| | KubeletConfiguration field | `rotateCertificates` | `serverTLSBootstrap` | | Signer | `kubernetes.io/kube-apiserver-client-kubelet` | `kubernetes.io/kubelet-serving` | | Auto-approved by kube-controller-manager | yes, after SubjectAccessReview | **no** | | Default without the field | bootstrap cert, no rotation | kubelet self-signs its own cert | Without `serverTLSBootstrap`, the kubelet generates a **self-signed** serving certificate. The API server connects to it anyway unless `--kubelet-certificate-authority` is set, so the API-server-to-kubelet hop is not really verified. Clients that do verify, such as metrics-server in its default configuration, reject it. With `serverTLSBootstrap: true` (the `RotateKubeletServerCertificate` feature gate, Beta and on by default since 1.12, must be enabled), the kubelet asks the `kubernetes.io/kubelet-serving` signer for a CA-signed certificate. Kubernetes deliberately **does not auto-approve** these requests. A serving certificate asserts DNS names and IP addresses, and the built-in approver has no reliable way to know that a node really owns the SANs it asked for. A compromised node could otherwise obtain a certificate for another node's address. So operators do one of the following: - Approve by hand: `kubectl get csr` then `kubectl certificate approve <name>`. - Run a custom approver that checks the SANs against the Node object's addresses. Until approval, the kubelet has no serving certificate at all, and `kubectl logs` against pods on that node fails with a TLS error. That is a common surprise the first time someone turns the field on. ## What to check when rotation breaks - `kubectl get csr` for pending or denied requests from `system:node:*`. - The kubelet's log for certificate manager errors. - The expiry of `kubelet-client-current.pem` with `openssl x509 -enddate`. - Whether kube-controller-manager is running. With no signer, CSRs stay approved but unsigned. ## Operating it across a fleet On a small cluster, approving serving CSRs by hand is manageable: 9 nodes produce 9 requests at join time and one each per rotation. It stops being manageable when nodes are replaced often. A few habits help: - Filter `kubectl get csr` by signer and requester so a pending `kubernetes.io/kubelet-serving` request stands out from routine client renewals. - Never approve a serving CSR without reading its SANs. An approval is a statement that this node owns those names and addresses. - Remember that approving is not signing. A CSR shows `Approved` but no certificate until kube-controller-manager's signer processes it. - Alert on CSRs that stay pending for more than a few minutes, because a node with no serving certificate quietly breaks `kubectl logs` for everything scheduled on it.