skip to content

Cluster Lifecycle

Standing a cluster up and keeping it alive: kubeadm bootstrap, control-plane-first upgrades and version skew, draining nodes for maintenance, renewing expiring certificates, and restoring after loss. It separates cluster users from cluster operators.

part ofKubernetesoverview, primer and where to startread it →
on this pageshow

questions

28

In a managed Kubernetes service, which parts of the cluster does the provider operate, and what stays the cluster owner's responsibility?

level: juniorimportance: must knowfreq 70%

answer

  1. split at the API endpoint
  2. behind it: apiserver, etcd, controllers
  3. in front of it: nodes, workloads, RBAC
  4. no SSH, no flags, no control-plane nodes
  5. managed node groups automate, not decide

basics

~10 s

The provider runs the control plane: kube-apiserver, etcd, the scheduler and controller managers, including their hosts, patching and availability. You still own worker nodes, workloads, RBAC, add-ons, network choices and deciding when to upgrade.

solid answer

~40 s

A managed Kubernetes service splits the cluster at the API endpoint. The provider operates everything behind it: the `kube-apiserver`, `etcd`, `kube-scheduler`, `kube-controller-manager` and usually the `cloud-controller-manager`, plus the machines they run on, their certificates, patch releases, etcd operation and a multi-replica, highly available setup backed by an SLA. Everything you *put into* the cluster is still yours: worker nodes (their OS image, size and replacement, unless you pay for a managed node group, which automates provisioning but not your capacity choices), workloads and their resource settings, RBAC, admission webhooks, CNI and add-on configuration, and scheduling the minor-version upgrades within the provider's window. The control-plane machines never appear in `kubectl get nodes`, and you cannot SSH to them or change their flags.

code

bash · 8 lines
bash
# Only worker nodes are listed; control-plane hosts are hidden
kubectl get nodes -o wide

# The API server still reports its own health
kubectl get --raw='/readyz?verbose'

# Server version is the provider-run control plane's version
kubectl version

go deeper

for a junior

Name the control-plane components the provider runs and list three things that stay yours: nodes, workloads and access control.

for a middle

Explain where the line sits (the API endpoint), what a managed node group automates and why control-plane nodes are invisible to kubectl.

for a senior

Show you can place an incident on the right side of the line quickly, such as a failing webhook versus an etcd problem, and know what evidence you can still gather.

for a principal

Frame the split as a trade: the provider absorbs control-plane toil and risk, and you give up configuration freedom and some diagnostic depth.

## What a managed control plane is A Kubernetes cluster has two halves. The **control plane** is the set of processes that store and reconcile the desired state: `kube-apiserver` (the only entry point), `etcd` (the key-value store holding every object), `kube-scheduler` (picks a node for each new Pod), `kube-controller-manager` (runs the built-in reconcile loops) and, on cloud infrastructure, `cloud-controller-manager` (load balancers, node addresses, routes). The **data plane** is the set of worker nodes running the kubelet, a container runtime and kube-proxy or its CNI replacement. In a **managed Kubernetes service**, the provider runs the control plane as a product. You receive an API endpoint and a kubeconfig; the processes behind that endpoint run on machines you never see. ## What the provider takes over | Area | Provider's job | |---|---| | Control-plane hosts | Provisioning, OS patching, replacement on failure | | `kube-apiserver` | Replicas behind a load balancer, TLS serving certificates, flag configuration | | `etcd` | Quorum sizing, disk, compaction and defragmentation, its own backups | | Patch releases | Rolling new patch versions onto the control plane, usually automatically | | Availability | Spreading replicas across failure domains, an SLA on the API endpoint | | Cluster PKI | Issuing and rotating the control-plane certificates | The practical effect is that the hardest day-2 work of a self-run cluster (etcd quorum loss, expiring control-plane certificates, a failed apiserver upgrade) becomes the provider's incident, not yours. ## What stays with you - **Workloads**: manifests, `resources.requests` and `resources.limits`, probes, PodDisruptionBudgets. - **Access control**: RBAC Roles and bindings, ServiceAccounts, and the mapping of cloud identities to Kubernetes users. - **Extensions you install**: admission webhooks, CRDs and operators, ingress or Gateway API implementations, a metrics pipeline. - **Networking choices**: the CNI configuration, NetworkPolicies, which Services are exposed. - **Upgrade timing**: the provider offers new minor versions, but you choose when to move within its support window. - **Worker nodes**, fully or partly, depending on how you run them. A useful rule: the provider guarantees that the API server answers; it does not guarantee that what you configured through it is correct. A validating admission webhook with `failurePolicy: Fail` whose backend is down will block writes, and that outage is yours. ## Managed node groups versus self-managed nodes | Aspect | Managed node group | Self-managed nodes | |---|---|---| | Provisioning and joining | Provider creates machines and joins them | You build the image and bootstrap the kubelet | | Node OS image | Provider-curated, updated on request | You patch and rebuild | | Node version upgrades | Provider drives a rolling replacement | You cordon, drain and replace | | Kubelet flags and config | Limited, through exposed options | Full control of `KubeletConfiguration` | | Capacity and instance choice | Still yours | Yours | A managed node group removes toil but not decisions: you still choose machine sizes, how many nodes, taints and labels, and whether your Pods tolerate a rolling replacement. Some services go further and hide nodes entirely, billing per Pod, which removes node choice as well. ## What you can no longer see or touch 1. **Control-plane nodes are absent** from `kubectl get nodes`, and there are no `kube-apiserver` static pods in `kube-system`. 2. **No SSH** to control-plane hosts and no direct `etcdctl` access. 3. **No component flags**: you cannot add `--feature-gates` or change `--audit-policy-file`; only the options the provider exposes. 4. **Logs arrive through the provider's logging integration**, if you enable it, rather than from files on disk. ## A worked example A team runs a recommendation-model inference server, each replica capped at a 2.6 GiB memory limit. When a replica is OOM-killed, that is the team's problem: the limit, the model size and the node size are all on their side of the line. When `kubectl apply` times out because an etcd member lost its disk, that is the provider's problem, and the team's job is to check the provider's status page and its support channel rather than log on to a machine they do not have. ## Why interviewers ask The question checks whether a candidate knows that "managed" describes one half of the cluster. Candidates who think a managed service also patches their images, fixes their RBAC or upgrades them forever without a decision usually have not run one.

  • Why does `kubectl get nodes` on a managed cluster list no control-plane nodes?
    The control-plane processes run on the provider's machines, usually outside your network, and those machines never register a Node object with your API server. Because they are not Nodes in your cluster, the scheduler never places your Pods on them. You see the control plane only through its API endpoint and whatever logs or metrics the provider forwards.
  • Your validating admission webhook's backend crashes and every Deployment update fails. Is that the provider's outage?
    No. The API server is healthy and doing exactly what your `ValidatingWebhookConfiguration` told it to do: with `failurePolicy: Fail`, an unreachable webhook rejects the request. The provider's SLA covers the endpoint answering, not the extensions you registered. Fixing it means restoring the webhook backend, narrowing its rules or selectors, or, as a last resort, deleting the configuration.
  • What does a managed node group change compared with nodes you bootstrap yourself?
    It automates creating machines from a provider-curated image, joining them to the cluster and replacing them during version upgrades. You still pick machine sizes, counts, labels and taints, and your workloads still need to survive the rolling replacement. In exchange you usually get less control over kubelet configuration and the node OS.

It is like renting a flat in a serviced building: the landlord keeps the lifts, boiler and wiring running, but what you plug in, who you give keys to and how you furnish the rooms are still your business.

saying these in an interview costs you the question

  • A managed service also patches the container images my workloads run.
  • The provider is responsible for my RBAC bindings being correct.
  • With a managed control plane I never have to plan a version upgrade.
  • Worker nodes are always part of what the provider manages.
  • I can SSH to the control-plane nodes to debug the API server.
  • If my admission webhook breaks the API, the provider's SLA covers it.
open as a page

In Kubernetes, what do `kubectl cordon`, `kubectl drain` and `kubectl uncordon` each do when you take a node out of service?

level: juniorimportance: must knowfreq 72%

basics

~20 s

Cordon marks a node unschedulable so no new pods land there. Drain cordons it and then evicts its pods so their controllers recreate them elsewhere. Uncordon makes the node schedulable again without moving any pods back.

open as a page

When a Kubernetes cluster is lost, what does an etcd snapshot bring back, and what must come from volume backups or Git instead?

level: middleimportance: must knowfreq 62%

basics

~20 s

An etcd snapshot restores Kubernetes API objects (Deployments, Secrets, RBAC, PVC and PV objects, custom resources) but none of the bytes stored on persistent volumes. Volume data needs CSI snapshots or Velero; Git re-creates only what was committed.

open as a page

When a node runs kubeadm join with a bootstrap token and --discovery-token-ca-cert-hash, what does each value prove, and to whom?

level: middleimportance: must knowfreq 66%

basics

~20 s

The bootstrap token is a short-lived shared secret: the node checks a token-signed cluster-info and the API server accepts the new kubelet. The CA cert hash pins the cluster CA public key, so a leaked token cannot impersonate the cluster.

open as a page

Why do organizations running Kubernetes split workloads across many clusters instead of one large cluster, and what does each driver buy them?

level: middleimportance: must knowfreq 62%

basics

~10 s

A Kubernetes cluster is one failure and control domain, so teams run many clusters to limit blast radius, place workloads in specific regions, draw compliance boundaries and give tenants isolation that namespaces cannot provide.

open as a page

In Kubernetes' version skew policy, how far may kubelets and kubectl drift from kube-apiserver, and why does that force control-plane-first upgrades?

level: middleimportance: must knowfreq 72%

basics

~20 s

Kubelets may be up to three minor versions older than kube-apiserver but never newer; kubectl may be one minor older or newer. Since nothing may be newer than the API server, it is upgraded first, then controllers and scheduler, then kubelets.

open as a page

A kubeadm-built Kubernetes cluster ran untouched for a year and kubectl now fails with 'x509: certificate has expired'. How do you diagnose and recover it?

level: seniorimportance: must knowfreq 62%

basics

~20 s

Confirm the expiry with kubeadm certs check-expiration, run kubeadm certs renew all on every control-plane node, restart the control-plane static pods so they load the new files, then copy the renewed admin.conf. Kubelets whose own certificate expired need re-bootstrapping.

open as a page

Your managed Kubernetes cluster's minor version is nearing the end of the provider's support window. What happens next, and how do you keep a sustainable upgrade cadence?

level: seniorimportance: must knowfreq 58%

basics

~20 s

Past the window the version stops getting patches, and most providers either upgrade the control plane for you or charge for extended support. Plan roughly three minor upgrades a year, rehearsed on a non-production cluster, before the provider forces one.

open as a page

Before upgrading a Kubernetes cluster, how do you find clients still calling an API version the target release removes, and what actually breaks afterwards?

level: seniorimportance: must knowfreq 56%

basics

~10 s

Watch the apiserver_requested_deprecated_apis metric and audit events annotated k8s.io/removed-release, and scan manifests. After the upgrade, stored objects survive, but every manifest, pipeline or controller still using the removed version fails.

open as a page

For local Kubernetes development, how do kind, minikube and k3s differ, and when would you choose each?

level: juniorimportance: should knowfreq 46%

basics

~20 s

kind runs each Kubernetes node as a container bootstrapped by kubeadm, suiting CI and multi-node tests; minikube runs a local cluster in a VM or container with an addon catalogue; k3s is a lightweight single-binary distribution also used on edge nodes.

open as a page

In a kubeadm-built Kubernetes cluster, why do the control-plane certificates expire after one year, and how do you check when they expire?

level: juniorimportance: should knowfreq 52%

basics

~20 s

kubeadm creates a cluster PKI in /etc/kubernetes/pki: CAs valid for ten years and leaf certificates valid for one year, so a stolen leaf is not useful for long. Running kubeadm certs check-expiration on a control-plane node shows each expiry date.

open as a page

How does a Kubernetes kubelet get and rotate its certificates through CertificateSigningRequests, and why do rotateCertificates and serverTLSBootstrap differ at approval?

level: middleimportance: should knowfreq 44%

basics

~20 s

The kubelet submits CertificateSigningRequests to the API server and kube-controller-manager signs them with the cluster CA. Client-certificate CSRs, used with rotateCertificates, are auto-approved. Serving-certificate CSRs, used with serverTLSBootstrap, are not, so someone must approve them.

open as a page

On a managed Kubernetes control plane, why can't you set kube-apiserver flags or enable alpha feature gates, and what do you use instead?

level: middleimportance: should knowfreq 42%

basics

~20 s

The provider owns the kube-apiserver command line and supports only tested, GA or beta behaviour across its fleet. You configure the cluster through API objects instead: admission policies, webhooks, CRDs, API Priority and Fairness objects and RBAC.

open as a page

In Cluster API, what is a management cluster, and how does it create and upgrade workload Kubernetes clusters declaratively?

level: middleimportance: should knowfreq 44%

basics

~20 s

A Cluster API management cluster is a Kubernetes cluster that runs Cluster API controllers. You declare workload clusters there as custom resources, and the controllers provision machines, bootstrap them with kubeadm and replace them to upgrade.

open as a page

When `kubectl drain` refuses to run, what do its `--ignore-daemonsets`, `--delete-emptydir-data` and `--force` flags each override, and what does each cost?

level: middleimportance: should knowfreq 58%

basics

~10 s

--ignore-daemonsets lets drain proceed while leaving DaemonSet pods running. --delete-emptydir-data accepts that emptyDir contents are deleted with their pods. --force removes pods that have no controller, which nothing will ever recreate.

open as a page

On a three-node kubeadm cluster where every node runs the control plane, what is the sequence to upgrade Kubernetes by one minor version?

level: middleimportance: should knowfreq 60%

basics

~20 s

Upgrade kubeadm on the first node, run kubeadm upgrade plan and kubeadm upgrade apply, then upgrade its kubelet; on each other node upgrade kubeadm, run kubeadm upgrade node, then its kubelet, draining one node at a time.

open as a page

After losing a Kubernetes control plane, you restore last night's etcd snapshot into a replacement one. What breaks once kube-apiserver serves that state, and how do you handle it?

level: seniorimportance: should knowfreq 44%

basics

~20 s

Restored state is old and bound to old identities. Reuse the original PKI and encryption keys, stop every API server before restoring, delete Node objects for vanished machines, restart controllers and kubelets, then reconcile drift caused by the rewind.

open as a page

How do you bootstrap a highly available Kubernetes control plane with kubeadm, and what must be decided before the first kubeadm init?

level: seniorimportance: should knowfreq 54%

basics

~20 s

Put a load balancer or DNS name in front of the future API servers, pass it as --control-plane-endpoint to the first kubeadm init, share certificates with --upload-certs, then run kubeadm join --control-plane on the others; choose stacked or external etcd up front.

open as a page

You must drain 5 nodes of a 64-node, two-zone Kubernetes cluster running a feature-flag evaluation service, and new nodes take 19 minutes to join. How do you run that maintenance safely?

level: seniorimportance: should knowfreq 40%

basics

~20 s

Confirm the other nodes can absorb the evicted pods' requests, adding capacity before the 19-minute gap bites. Cordon the whole batch, drain one node at a time with a timeout, wait until the service is whole, then continue.

open as a page

You own disaster recovery for a 140-node Kubernetes cluster shared by 22 product teams. How do you split recovery between etcd snapshots, Velero and Git re-apply, and prove it works?

level: principalimportance: should knowfreq 38%

basics

~20 s

Match each failure to a recovery unit: rebuild plus Git re-apply for stateless workloads, Velero with CSI snapshots for namespaces with data or objects missing from Git, and etcd snapshots for rewinding the whole control plane. Prove it with timed restore drills.

open as a page

A retailer plans a 5-node edge Kubernetes cluster in each of 400 stores. Would you use provider-managed control planes or run your own, and how would you decide?

level: principalimportance: should knowfreq 34%

basics

~20 s

Weigh the per-cluster fee times 400, the store's dependence on a WAN link to a regional control plane, and the memory a local control plane takes from five small nodes. Many fleets self-run a lightweight distribution with strong fleet automation instead.

open as a page

Your Kubernetes platform is one 38-node cluster with mixed spot and on-demand pools, and its loyalty-points accrual namespace alone runs 1,180 pods. How would you decide whether to split it into several clusters?

level: principalimportance: should knowfreq 46%

basics

~20 s

Split only when a concrete driver (blast radius, region, compliance scope or hard tenancy) outweighs the extra control planes, add-on stacks, upgrades and stranded capacity, and only once fleet tooling can run many clusters as consistently as one.

open as a page

Your clinical-records portal runs on a three-node on-prem kubeadm cluster two minor releases behind; how do you choose between in-place upgrades and a replacement cluster, and set a cadence?

level: principalimportance: should knowfreq 40%

basics

~20 s

Two minors behind usually means two in-place kubeadm hops, rehearsed first, if the portal fits on two nodes; a replacement cluster pays off when spare hardware exists, rollback matters most, or the gap is larger. Then upgrade every minor.

open as a page

Can a kubeadm-built Kubernetes cluster on v1.35 be upgraded straight to v1.37, and if not, why not?

level: juniorimportance: nice to knowfreq 38%

basics

~10 s

No. A Kubernetes control plane is upgraded one minor version at a time, so v1.35 goes to v1.36 and then to v1.37, and kubeadm upgrade apply rejects a two-minor jump for a released version.

open as a page

In a Velero restore of a Kubernetes namespace whose PVCs were backed up as CSI VolumeSnapshots, in what order are objects recreated, and how do PVCs get volumes again?

level: seniorimportance: nice to knowfreq 33%

basics

~20 s

Velero restores by priority: CRDs, namespaces, storage and snapshot classes, snapshot objects, PVs and PVCs, RBAC, Secrets and ConfigMaps, then Pods and controllers. CSI-backed PVs are not restored; each PVC gets a dataSource pointing at its VolumeSnapshot and is dynamically provisioned.

open as a page

After editing kube-apiserver.yaml in /etc/kubernetes/manifests on a kubeadm control-plane node, kubectl stops responding. How do you diagnose and recover?

level: seniorimportance: nice to knowfreq 34%

basics

~20 s

The kubelet restarts the API server from the edited static pod manifest, so kubectl is useless; debug on the node with crictl ps -a, crictl logs and the kubelet journal, fix the file or restore a backup kept outside the manifests directory.

open as a page

Why does kubeadm certs renew never replace a Kubernetes cluster's CA, and how would you rotate the cluster CA without an outage?

level: seniorimportance: nice to knowfreq 26%

basics

~20 s

Replacing a Kubernetes cluster CA changes what every component trusts, so kubeadm refuses to do it automatically. Rotating it without an outage means trusting old and new CAs together, re-issuing every leaf from the new CA, then removing the old CA.

open as a page

With the Kubernetes Multi-Cluster Services API, how would you expose a loyalty-points accrual service running in two clusters under one name to a third cluster?

level: seniorimportance: nice to knowfreq 30%

basics

~10 s

Create a ServiceExport with the Service's name and namespace in both clusters that run it. An MCS implementation then creates a matching ServiceImport across the ClusterSet, and callers use name.namespace.svc.clusterset.local.

open as a page