skip to content

GPU Scheduling on Kubernetes

The GPU Operator, MIG and time-slicing turn GPUs into schedulable slices, and Kueue or Volcano admit a multi-pod job all at once or not at all. ML-platform rounds probe the half-scheduled cluster.

on this pageshow

explore

questions

6

For notebook users on 80 GB GPUs managed by the NVIDIA GPU Operator, when do you pick MIG slices over time-sliced replicas, and what does each isolate?

level: middleimportance: must knowfreq 18%

answer

  1. hardware partition versus shared clock
  2. memory and fault isolation
  3. replicas in the sharing config
  4. renameByDefault and the .shared suffix
  5. failRequestsGreaterThanOne

basics

~20 s

MIG splits a GPU into hardware instances with their own memory and fault isolation, in fixed profiles like 1g.10gb. Time-slicing hands out replicas of one GPU that share its memory and compute, isolating nothing. Pick MIG when tenants must not disturb each other.

solid answer

~40 s

On an 80 GB H100, MIG offers fixed profiles from `1g.10gb` (up to seven per GPU) to `7g.80gb`; each instance has its own memory and compute slice, so one notebook's out-of-memory error or crash stays inside its instance. Time-slicing lives in the device plugin's sharing config: `replicas` multiplies what the node advertises, `renameByDefault: true` renames the resource to `nvidia.com/gpu.shared`, and `failRequestsGreaterThanOne: true` rejects requests for more than one replica, because extra replicas buy no extra compute. All replicas share the GPU's memory and one fault domain. So MIG suits many light, untrusted notebooks on MIG-capable parts; time-slicing suits more than seven tenants per GPU, GPUs without MIG, or cooperating workloads. The two combine: you can time-slice a MIG resource under the mixed strategy.

code

yaml · 17 lines
yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: notebook-sharing
  namespace: gpu-operator
data:
  notebook: |-
    version: v1
    flags:
      migStrategy: none
    sharing:
      timeSlicing:
        renameByDefault: true
        failRequestsGreaterThanOne: true
        resources:
        - name: nvidia.com/gpu
          replicas: 4

go deeper

for a junior

Recall the one-line difference: MIG is a hardware partition with isolated memory and faults, time-slicing is shared time on a whole GPU with no isolation.

for a middle

Explain replicas, renameByDefault and failRequestsGreaterThanOne, compute what a node advertises, and name the 80 GB MIG profiles a notebook would use.

for a senior

Decide per pool from isolation needs, tenants per GPU and GPU generation, and account for lost per-container DCGM metrics and MIG relayouts evicting GPU pods.

for a principal

Frame sharing policy as tenant density against blast radius, including chargeback, noisy neighbours and how often layouts must change as demand shifts.

## Two ways to share one GPU Notebook users rarely need a whole 80 GB GPU, yet Kubernetes counts GPUs as whole units, so sharing has to happen below the scheduler. Either the GPU is cut into smaller real devices (**MIG**, Multi-Instance GPU), or the device plugin advertises one GPU several times and lets the processes take turns (**time-slicing**). The NVIDIA GPU Operator manages both. | | MIG | Time-slicing | |---|---|---| | Mechanism | Hardware partition into GPU instances | Device plugin advertises N replicas of one GPU; CUDA interleaves processes | | Memory | Each instance owns a fixed fraction | Every replica can use all of the GPU's memory | | Fault isolation | Yes, at the hardware layer | None: one fault domain | | Granularity | Fixed profiles per GPU model | Any `replicas` count you set | | GPU support | MIG-capable parts only | Any NVIDIA GPU, including older ones | | Configured by | MIG Manager, via the `nvidia.com/mig.config` node label | The device plugin's `sharing.timeSlicing` config | | Resource name | `nvidia.com/gpu` (single strategy) or `nvidia.com/mig-<profile>` (mixed) | `nvidia.com/gpu`, or `nvidia.com/gpu.shared` with `renameByDefault` | ## MIG profiles on an 80 GB H100 The MIG user guide lists these GPU instance profiles for the H100 80 GB: - `1g.10gb`: 1/8 of memory, 1/7 of the SMs, up to **7** per GPU - `1g.20gb`: up to 4 per GPU - `2g.20gb`: up to 3 per GPU - `3g.40gb`: up to 2 per GPU - `4g.40gb`: 1 per GPU - `7g.80gb`: the whole GPU, 1 per GPU Other parts use other names (a 40 GB A100 starts at `1g.5gb`), so check the table for the GPU you own. The sizes are fixed: a notebook that needs 15 GB has to be given a `1g.20gb` or `2g.20gb` instance, because MIG cannot carve 15 GB. ## Time-slicing through the device plugin - **`replicas`** under `sharing.timeSlicing.resources` makes the plugin hand out that many references to each GPU: eight GPUs with `replicas: 4` become 32 advertised resources. - **`renameByDefault`** (default `false`): when `true`, the resource is advertised as `nvidia.com/gpu.shared`, so a pod opts into sharing by name. When `false`, GPU Feature Discovery appends `-SHARED` to `nvidia.com/gpu.product` so a node selector can still tell shared nodes apart. - **`failRequestsGreaterThanOne`** (default `false` for backward compatibility, recommended `true`): a container asking for more than one shared replica fails with `UnexpectedAdmissionError` and must be deleted, fixed and redeployed. - CUDA gives an **equal share of time to every GPU process** across all pods, and each pod may run as many processes as it likes. A pod that requests two replicas does not get twice the compute, and nothing stops one notebook from allocating most of the 80 GB. - The config is a ConfigMap referenced from the ClusterPolicy's `spec.devicePlugin.config`; a `default` key applies it cluster-wide, and the node label `nvidia.com/device-plugin.config` selects a named config per node. The same method applies to all GPUs on a node. ## Choosing for the notebook pool Say four of the 32 nodes, 32 GPUs, serve notebooks: - **MIG, `all-1g.10gb`**: 32 x 7 = **224** isolated 10 GB instances. - **Time-slicing, `replicas: 4`**: 32 x 4 = **128** shared replicas, each able to touch all 80 GB. 1. Many users who do not trust each other, with memory needs that fit a profile: **MIG**. 2. GPUs without MIG, more than seven tenants per GPU, or one team's cooperating jobs: **time-slicing**. 3. Both: time-slicing also accepts the resources the mixed MIG strategy creates, so `nvidia.com/mig-1g.10gb` can carry `replicas` too. Instances stay isolated from each other; pods inside one instance share it. ## What a notebook pod asks for The request in the pod spec follows from the choice, which is why notebook images and templates must be written for one pool: - **MIG, single strategy**: `nvidia.com/gpu: 1`, plus a node selector on `nvidia.com/gpu.product`, which GPU Feature Discovery suffixes with the profile (for example `-MIG-1g.10gb`), so the pod lands on the sliced nodes. - **MIG, mixed strategy**: the profile is in the resource name, `nvidia.com/mig-1g.10gb: 1`. - **Time-slicing with `renameByDefault: true`**: `nvidia.com/gpu.shared: 1`, which no whole-GPU training pod will ever request by accident. - **Time-slicing with the default `false`**: `nvidia.com/gpu: 1` plus a node selector on the `-SHARED` product label; without it, the notebook could take a whole GPU on a training node. ## Operational caveats - The DCGM exporter cannot associate metrics with containers when time-slicing is on, which makes per-notebook accounting harder. - The operator does not watch the time-slicing ConfigMap for edits; updating it needs the documented reapply step. - Changing a node's MIG layout terminates the GPU pods on that node, so relayouts are maintenance events, not a per-notebook knob.

  • Why turn on failRequestsGreaterThanOne in the device plugin's time-slicing config?
    It defaults to `false` for backward compatibility. Set to `true`, a container that asks for more than one shared replica fails with `UnexpectedAdmissionError`, which teaches users that a replica is an access ticket, not a fraction of compute. The pod then has to be deleted, fixed and redeployed.
  • Can a MIG instance itself be time-sliced?
    Yes. Time-slicing supports `nvidia.com/gpu` and the resources the mixed MIG strategy creates, so `nvidia.com/mig-1g.10gb` can carry `replicas`. Instances stay isolated from each other, while the pods sharing one instance share its memory and its faults.

MIG is a building divided into flats, each with its own walls and meter; time-slicing is one shared flat where housemates take turns at a single kitchen. More people fit in the shared flat, but one housemate's mess stops everyone's dinner.

  • Time-slicing gives every replica its own share of the GPU's memory.
  • Requesting two time-sliced replicas doubles a pod's compute share.
  • MIG can carve an 80 GB GPU into whatever size a pod requests.
  • A notebook crashing on a time-sliced GPU affects only itself.
  • MIG and time-slicing can never be used on the same GPU.

Assumes NVIDIA GPU Operator 26.7 with device plugin v0.20.1 and the H100 80 GB MIG profile table; other GPU models offer different profiles.

open as a page

A 16-pod pretraining job on a shared Kubernetes GPU cluster has 11 pods running and holding GPUs while 5 stay Pending; how do Kueue and Volcano each prevent this?

level: seniorimportance: must knowfreq 13%

basics

~20 s

Kueue admits a Workload only when quota for every pod is free, can check node placement with topology-aware scheduling, and with waitForPodsReady evicts and requeues a Workload not fully ready in time. Volcano binds none of a PodGroup's pods until minMember can be placed.

open as a page

What does the NVIDIA GPU Operator install and manage on a Kubernetes cluster's GPU nodes, and which resource holds its configuration?

level: juniorimportance: should knowfreq 15%

basics

~20 s

The NVIDIA GPU Operator runs the whole GPU software stack as containers on each GPU node: driver, container toolkit, device plugin, GPU Feature Discovery, DCGM exporter, MIG Manager and a validator. One ClusterPolicy resource declares that desired state.

open as a page

In Kueue 0.20, how do a ResourceFlavor, a ClusterQueue and a LocalQueue decide when a team's GPU job is admitted, and how does a cohort let it borrow?

level: middleimportance: should knowfreq 12%

basics

~20 s

Kueue admits whole jobs against quota: a ResourceFlavor names a class of nodes, a ClusterQueue holds nominalQuota per flavor and resource, and a LocalQueue routes a namespace's jobs to it. ClusterQueues sharing a cohortName can borrow each other's unused quota.

open as a page

With the NVIDIA GPU Operator, how do the MIG single and mixed strategies change the resource a pod requests, and how do you repartition a node?

level: middleimportance: should knowfreq 8%

basics

~20 s

Under the single strategy every GPU on a node has the same MIG profile and each instance is advertised as nvidia.com/gpu; under mixed, each profile becomes its own resource such as nvidia.com/mig-3g.40gb. Repartition by changing the node's nvidia.com/mig.config label.

open as a page

On a Kubernetes GPU cluster wired in racks and blocks, how do Kueue topology-aware scheduling and Volcano network-topology scheduling keep a multi-node training job's pods close together?

level: seniorimportance: nice to knowfreq 5%

basics

~20 s

Kueue reads a Topology of node-label levels through a ResourceFlavor and, given podset-required-topology or podset-preferred-topology on the pod template, admits a job only where its pods fit one rack or block. Volcano models switches as HyperNode tiers and honours a job's networkTopology.

open as a page