What does the NVIDIA GPU Operator install and manage on a Kubernetes cluster's GPU nodes, and which resource holds its configuration?
answer
- one operator, many node daemons
- driver plus container toolkit
- labels, metrics and MIG layouts
- ClusterPolicy holds the desired state
- driver.enabled=false on preinstalled hosts
basics
~20 sThe NVIDIA GPU Operator runs the whole GPU software stack as containers on each GPU node: driver, container toolkit, device plugin, GPU Feature Discovery, DCGM exporter, MIG Manager and a validator. One ClusterPolicy resource declares that desired state.
solid answer
~40 sThe GPU Operator turns GPU node setup into a Kubernetes reconcile loop. Its Helm chart creates a `ClusterPolicy` (`clusterPolicy.deployCR: true` by default), and the operator deploys, on every GPU node, the NVIDIA driver container, the NVIDIA Container Toolkit, the device plugin that advertises the GPUs, GPU Feature Discovery for labels such as `nvidia.com/gpu.product`, the DCGM exporter for metrics, MIG Manager for MIG layouts and a validator; Node Feature Discovery comes along to label nodes by their hardware. In GPU Operator 26.7 each of these is on by default except standalone `dcgm`, because the exporter runs an embedded host engine. On hosts that already carry a driver you install with `driver.enabled=false`, and `kubectl get clusterpolicy` reporting `ready` is the health check.
go deeper
Name the layers the operator manages: driver, container toolkit, device plugin, GPU Feature Discovery, DCGM exporter and MIG Manager, and say that one ClusterPolicy declares them.
Explain which Helm values switch components off, why driver.enabled=false matters on preinstalled hosts, and which node labels GPU Feature Discovery writes for later selection.
Show you operate it: ClusterPolicy readiness as a rollout gate, patching ClusterPolicy fields for MIG strategy or sharing, and knowing time-slicing breaks per-container DCGM attribution.
Weigh operator-managed drivers against drivers baked into node images: upgrade cadence, mixed GPU generations and who owns driver rollouts across a shared fleet.
## Why GPU nodes get an operator Before a pod can use a GPU, a node needs several layers in place: a kernel driver that matches the GPU, a container-runtime hook that mounts the GPU devices and driver libraries into containers, a **device plugin** that advertises GPUs to the kubelet as a schedulable resource, labels that describe the hardware, and monitoring. Installing and upgrading those by hand on 32 nodes with eight 80 GB GPUs each drifts quickly: one node gets a newer driver, another misses the runtime hook, and pods fail in ways that look like application bugs. The **NVIDIA GPU Operator** packages each layer as a container image and manages them the Kubernetes way: one custom resource states the desired stack, and the operator keeps every GPU node, including ones added later, converged on it. ## What runs on each GPU node | Component | What it does | GPU Operator 26.7 Helm value | |---|---|---| | NVIDIA driver container | Installs and loads the kernel driver on the host | `driver.enabled: true` | | NVIDIA Container Toolkit | Configures the container runtime so containers receive GPU devices and libraries | `toolkit.enabled: true` | | NVIDIA device plugin | Advertises the node's GPUs, MIG instances or time-sliced replicas to the kubelet | `devicePlugin.enabled: true` | | GPU Feature Discovery (GFD) | Labels nodes, e.g. `nvidia.com/gpu.product`, `nvidia.com/gpu.count`, `nvidia.com/mig.strategy` | `gfd.enabled: true` | | DCGM exporter | Exposes GPU metrics for Prometheus | `dcgmExporter.enabled: true` | | Standalone DCGM | A separate host engine | `dcgm.enabled: false` (the exporter uses its embedded one) | | MIG Manager | Applies the MIG layout a node's `nvidia.com/mig.config` label names | `migManager.enabled: true` | | Validator | Checks that the stack works on the node | part of the stack | | Node Feature Discovery | Labels nodes with hardware features | `nfd.enabled: true` | The device plugin and GFD ship from the same project (`k8s-device-plugin` v0.20.1 in this release). How the device-plugin API itself registers resources with the kubelet is a Kubernetes topic; here it matters only that the operator installs and configures it. ## ClusterPolicy: one desired state - The Helm chart creates a **`ClusterPolicy`** when `clusterPolicy.deployCR` is `true`, the default. Every component above is a section of its spec. - Changes go through that object. The documented way to switch the MIG strategy is a patch of `/spec/mig/strategy` on `clusterpolicies.nvidia.com/cluster-policy`; a time-slicing config is attached through `spec.devicePlugin.config`. - `kubectl get clusterpolicy` shows whether the stack has reached the `ready` state, which is the check to gate a rollout on. - Release 26.7 also carries an experimental stack built on Dynamic Resource Allocation, selected with `gpuCluster.deployCR=true` and `clusterPolicy.deployCR=false`; having both custom resources at once is an invalid configuration. ## Defaults worth knowing 1. **Preinstalled drivers.** Install with `--set driver.enabled=false`, and add `--set toolkit.enabled=false` when the container toolkit is preinstalled too. If you forget the first, the driver pod's init container detects the host driver and labels the node, so the driver pod terminates and is not rescheduled there. 2. **MIG is configured, not automatic.** `mig.strategy` defaults to `single`, but no GPU is partitioned until a node's `nvidia.com/mig.config` label asks MIG Manager for a layout. 3. **No sharing by default.** The device plugin advertises whole GPUs until you hand it a time-slicing config. 4. **Monitoring is included.** The DCGM exporter is on; it cannot attribute metrics to individual containers when time-slicing is enabled, which matters for per-team chargeback. ## Reading a node after install A quick health read on one GPU node, once the `ClusterPolicy` is `ready`: - `kubectl get pods -n gpu-operator` lists the per-node DaemonSet pods, among them `nvidia-driver-daemonset`, `nvidia-device-plugin-daemonset`, `gpu-feature-discovery` and, on MIG-capable nodes, `nvidia-mig-manager`. - The node's capacity shows `nvidia.com/gpu: 8`, which is the device plugin doing its job. - GPU Feature Discovery's labels read `nvidia.com/gpu.present=true`, `nvidia.com/gpu.count=8` and `nvidia.com/gpu.product=NVIDIA-H100-80GB-HBM3`; schedulers and node selectors build on these. - If a pod later fails to see its GPU, check these in order: driver pod, toolkit pod, device plugin pod. The layers depend on each other in that order. ## What the operator does not do - It does **not schedule** pods: kube-scheduler places them, and queueing or gang admission comes from Kueue or Volcano. - It does **not decide who gets a GPU**: requests, taints and quotas are configured elsewhere. - It does **not choose a partitioning** for you: MIG geometry and time-slicing replicas are your configuration, which it then applies. On the 32-node cluster the operator's first job is unglamorous: bring every node to the point where the device plugin advertises eight GPUs and GFD labels them `NVIDIA-H100-80GB-HBM3`. Slicing the notebook nodes and queueing the training teams are built on top of that.
- What happens if the GPU Operator's driver component is left enabled on nodes that already have a host driver?The driver pod's init container detects the preinstalled driver and labels the node, so the driver pod terminates and is not rescheduled there. Passing `driver.enabled=false` at install is the cleaner path, plus `toolkit.enabled=false` when the NVIDIA Container Toolkit is also preinstalled on the hosts.
- Why is the standalone dcgm component off by default while the DCGM exporter is on?In GPU Operator 26.7's Helm values the exporter runs an embedded `nv-hostengine`, so a separate DCGM pod is optional; `dcgm.enabled: true` deploys one. Either way, the exporter cannot attribute metrics to containers when device-plugin time-slicing is enabled.
- The GPU Operator replaces kube-scheduler for pods that request GPUs.
- The operator cannot install drivers, so every node needs a hand-installed driver first.
- Installing the GPU Operator partitions every GPU with MIG automatically.
- GPU metrics need a separate monitoring install because the operator ships none.
Assumes NVIDIA GPU Operator 26.7 (v26.7.1) with device plugin and GPU Feature Discovery v0.20.1; component toggles and defaults are those of its Helm values.