skip to content

Cluster Bootstrap

7 roadmaps4 questions

How a cluster comes into being: kubeadm init and join with bootstrap tokens, control-plane static pods on disk, stacked or external etcd, and alternatives like kind, minikube and k3s. Standing up an HA cluster is a standard self-managed opener.

part ofKubernetesoverview, primer and where to startread it →
on this pageshow

explore

questions

4

When a node runs kubeadm join with a bootstrap token and --discovery-token-ca-cert-hash, what does each value prove, and to whom?

level: middleimportance: must knowfreq 66%

answer

  1. two directions of trust
  2. shared secret versus pinned key
  3. signed ConfigMap in kube-public
  4. sha256 of Subject Public Key Info
  5. tokens expire after a day

basics

~20 s

The bootstrap token is a short-lived shared secret: the node checks a token-signed cluster-info and the API server accepts the new kubelet. The CA cert hash pins the cluster CA public key, so a leaked token cannot impersonate the cluster.

solid answer

~40 s

`kubeadm join` needs mutual trust. The **bootstrap token** (`[a-z0-9]{6}.[a-z0-9]{16}`, stored as a `bootstrap-token-<id>` Secret in `kube-system`, 24h TTL by default) works in both directions: the node reads the `cluster-info` ConfigMap in `kube-public`, which carries the CA and API address and is JWS-signed with the token, and then uses the token as a bearer credential, authenticating as `system:bootstrap:<id>` to request its kubelet client certificate. The **`--discovery-token-ca-cert-hash`** (`sha256:` of the CA's Subject Public Key Info) pins the CA itself, so an attacker who knows the token cannot pose as the cluster. Skipping it with `--discovery-token-unsafe-skip-ca-verification` gives up that protection. Expired tokens are the classic late-join failure: mint a new one with `kubeadm token create --print-join-command`.

code

bash · 5 lines
bash
kubeadm token create --ttl 2h --print-join-command

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
  | openssl rsa -pubin -outform der 2>/dev/null \
  | openssl dgst -sha256 -hex | sed 's/^.* //'

go deeper

for a junior

Recall that a join needs a token and a CA hash, and that the token is short-lived. Know that kubeadm token create can print a new join command.

for a middle

Explain both directions of trust: the JWS-signed cluster-info and bearer auth use the token, while the hash pins the CA public key.

for a senior

Diagnose a join that fails days after init, and design token issuance for autoscaled nodes without falling back to non-expiring tokens.

for a principal

Weigh token-based bootstrap against file-based discovery or a machine-identity-driven node lifecycle, and set policy for where join credentials may live.

## The trust problem kubeadm join solves A fresh machine that wants to become a Kubernetes node knows almost nothing: an API server address and whatever the operator pasted on its command line. Two questions must be answered before it can run pods: 1. **Is this really my cluster?** The node must not send its identity to an impostor API server. 2. **Is this really a node I should admit?** The API server must not hand a kubelet identity to anyone who asks. `kubeadm join <endpoint> --token <token> --discovery-token-ca-cert-hash sha256:<hex>` answers both with two different pieces of data. ## The bootstrap token A **bootstrap token** is a short shared secret of the form `abcdef.0123456789abcdef`: a 6-character public **token ID** and a 16-character **token secret**, both lowercase letters and digits. - It is stored in the cluster as a Secret named `bootstrap-token-<token-id>` in `kube-system`, with type `bootstrap.kubernetes.io/token`. - `kubeadm init` creates one; `kubeadm token create` makes more. The default lifetime is **24 hours**; `--ttl 0` makes a token that never expires, which is convenient and risky. - kubeadm enables two kube-controller-manager controllers that ordinary clusters leave off: `bootstrapsigner` and `tokencleaner` (it sets `--controllers=*,bootstrapsigner,tokencleaner`). The second deletes expired token Secrets. The token is used twice. **Discovery.** The node anonymously reads the `cluster-info` ConfigMap in `kube-public`. kubeadm grants `system:anonymous` read access to exactly that object. It holds a kubeconfig with the API server address and the cluster CA certificate. The `bootstrapsigner` controller adds a JWS signature for each token under a key named `jws-kubeconfig-<token-id>`, computed with the token secret. The node verifies that signature, which proves the server side **knows the token**. **TLS bootstrap.** The kubelet then presents the token as a bearer credential. The API server authenticates it as user `system:bootstrap:<token-id>` in the group `system:bootstrappers:kubeadm:default-node-token`, and RBAC lets that group submit a CertificateSigningRequest for a kubelet client certificate. Once the certificate is issued, the kubelet stops using the token and talks with its own identity. How that CSR is signed, approved and later rotated is certificate-lifecycle material, not bootstrap. ## The CA certificate hash A token proves only that the other side knows a shared secret. Tokens get pasted into tickets, CI logs and cloud-init user data, so assume one can leak. Someone holding it could run a fake API server, sign a fake `cluster-info`, and receive the node's traffic. **`--discovery-token-ca-cert-hash`** closes that gap. It is `sha256:` followed by the hex SHA-256 of the cluster CA's **Subject Public Key Info**, not of the whole certificate file. After discovery, kubeadm hashes the public key of the CA it received and refuses to continue on a mismatch. This is **public key pinning**: the value is not secret, but an attacker cannot forge it without the CA private key. | Value | Secret? | Proves | Verified by | |---|---|---|---| | token ID + secret | yes | caller knows the shared secret | node (JWS on `cluster-info`) and API server (bearer auth) | | CA cert hash | no | cluster CA is the expected one | node only | `--discovery-token-unsafe-skip-ca-verification` turns the pin off and leaves only token-based trust. Alternatives are file-based discovery (`--discovery-file` with a kubeconfig that already contains the CA). ## The late-join failure on an autoscaled cluster Consider **an 80-node cluster that autoscales between 20 and 80 nodes**, hosting **a multiplayer game-session backend**. The provisioning template baked in the join command from `kubeadm init`. For the first day, scale-ups work. Thirty-one hours later a player surge triggers scale-up and new machines never register: the token Secret was removed by `tokencleaner` after its 24-hour TTL, so discovery cannot find a matching JWS signature and the join fails. Fixes, from quick to durable: - `kubeadm token create --print-join-command` prints a fresh, complete join command, including the current CA hash. - Have the node-provisioning pipeline mint a short-lived token per scale-up instead of baking one in. - Avoid `--ttl 0` tokens as a shortcut; a leaked non-expiring token plus a skipped CA pin lets an attacker register nodes indefinitely. ## Computing the hash yourself The hash is printed by `kubeadm init`, but it can be recomputed from `/etc/kubernetes/pki/ca.crt` on any control-plane node, as the code example shows for an RSA CA key.

  • Why is the CA hash safe to paste into a public runbook while the token is not?
    The hash is a fingerprint of the CA public key. Knowing it lets you check a CA, not impersonate one, because forging a matching CA needs the private key. The token secret is a credential: it authenticates as a bootstrapper and lets its holder sign a fake `cluster-info`.
  • What identity does the kubelet use once kubeadm join finishes?
    It uses its own client certificate, issued through the CSR it filed while authenticated as `system:bootstrap:<token-id>`. From then on it authenticates as `system:node:<node-name>` in `system:nodes`, and the bootstrap token is no longer needed by that node.
  • How would you stop scale-up joins from failing when the original token expires?
    Do not bake the `kubeadm init` join command into the machine template. Have provisioning call `kubeadm token create --print-join-command` (or the API equivalent) per scale-up with a short TTL, or use a node lifecycle tool that manages tokens for you.

saying these in an interview costs you the question

  • The CA cert hash is a secret that must be protected like the token
  • The bootstrap token is the kubelet's permanent credential
  • Bootstrap tokens never expire unless someone deletes them
  • Skipping CA verification is harmless inside a private network
  • The CA hash is the SHA-256 of the whole ca.crt file
open as a page

For local Kubernetes development, how do kind, minikube and k3s differ, and when would you choose each?

level: juniorimportance: should knowfreq 46%

basics

~20 s

kind runs each Kubernetes node as a container bootstrapped by kubeadm, suiting CI and multi-node tests; minikube runs a local cluster in a VM or container with an addon catalogue; k3s is a lightweight single-binary distribution also used on edge nodes.

open as a page

How do you bootstrap a highly available Kubernetes control plane with kubeadm, and what must be decided before the first kubeadm init?

level: seniorimportance: should knowfreq 54%

basics

~20 s

Put a load balancer or DNS name in front of the future API servers, pass it as --control-plane-endpoint to the first kubeadm init, share certificates with --upload-certs, then run kubeadm join --control-plane on the others; choose stacked or external etcd up front.

open as a page

After editing kube-apiserver.yaml in /etc/kubernetes/manifests on a kubeadm control-plane node, kubectl stops responding. How do you diagnose and recover?

level: seniorimportance: nice to knowfreq 34%

basics

~20 s

The kubelet restarts the API server from the edited static pod manifest, so kubectl is useless; debug on the node with crictl ps -a, crictl logs and the kubelet journal, fix the file or restore a backup kept outside the manifests directory.

open as a page