For notebook users on 80 GB GPUs managed by the NVIDIA GPU Operator, when do you pick MIG slices over time-sliced replicas, and what does each isolate?
answer
- hardware partition versus shared clock
- memory and fault isolation
- replicas in the sharing config
- renameByDefault and the .shared suffix
- failRequestsGreaterThanOne
basics
~20 sMIG splits a GPU into hardware instances with their own memory and fault isolation, in fixed profiles like 1g.10gb. Time-slicing hands out replicas of one GPU that share its memory and compute, isolating nothing. Pick MIG when tenants must not disturb each other.
solid answer
~40 sOn an 80 GB H100, MIG offers fixed profiles from `1g.10gb` (up to seven per GPU) to `7g.80gb`; each instance has its own memory and compute slice, so one notebook's out-of-memory error or crash stays inside its instance. Time-slicing lives in the device plugin's sharing config: `replicas` multiplies what the node advertises, `renameByDefault: true` renames the resource to `nvidia.com/gpu.shared`, and `failRequestsGreaterThanOne: true` rejects requests for more than one replica, because extra replicas buy no extra compute. All replicas share the GPU's memory and one fault domain. So MIG suits many light, untrusted notebooks on MIG-capable parts; time-slicing suits more than seven tenants per GPU, GPUs without MIG, or cooperating workloads. The two combine: you can time-slice a MIG resource under the mixed strategy.
code
yaml · 17 linesapiVersion: v1
kind: ConfigMap
metadata:
name: notebook-sharing
namespace: gpu-operator
data:
notebook: |-
version: v1
flags:
migStrategy: none
sharing:
timeSlicing:
renameByDefault: true
failRequestsGreaterThanOne: true
resources:
- name: nvidia.com/gpu
replicas: 4go deeper
Recall the one-line difference: MIG is a hardware partition with isolated memory and faults, time-slicing is shared time on a whole GPU with no isolation.
Explain replicas, renameByDefault and failRequestsGreaterThanOne, compute what a node advertises, and name the 80 GB MIG profiles a notebook would use.
Decide per pool from isolation needs, tenants per GPU and GPU generation, and account for lost per-container DCGM metrics and MIG relayouts evicting GPU pods.
Frame sharing policy as tenant density against blast radius, including chargeback, noisy neighbours and how often layouts must change as demand shifts.
## Two ways to share one GPU Notebook users rarely need a whole 80 GB GPU, yet Kubernetes counts GPUs as whole units, so sharing has to happen below the scheduler. Either the GPU is cut into smaller real devices (**MIG**, Multi-Instance GPU), or the device plugin advertises one GPU several times and lets the processes take turns (**time-slicing**). The NVIDIA GPU Operator manages both. | | MIG | Time-slicing | |---|---|---| | Mechanism | Hardware partition into GPU instances | Device plugin advertises N replicas of one GPU; CUDA interleaves processes | | Memory | Each instance owns a fixed fraction | Every replica can use all of the GPU's memory | | Fault isolation | Yes, at the hardware layer | None: one fault domain | | Granularity | Fixed profiles per GPU model | Any `replicas` count you set | | GPU support | MIG-capable parts only | Any NVIDIA GPU, including older ones | | Configured by | MIG Manager, via the `nvidia.com/mig.config` node label | The device plugin's `sharing.timeSlicing` config | | Resource name | `nvidia.com/gpu` (single strategy) or `nvidia.com/mig-<profile>` (mixed) | `nvidia.com/gpu`, or `nvidia.com/gpu.shared` with `renameByDefault` | ## MIG profiles on an 80 GB H100 The MIG user guide lists these GPU instance profiles for the H100 80 GB: - `1g.10gb`: 1/8 of memory, 1/7 of the SMs, up to **7** per GPU - `1g.20gb`: up to 4 per GPU - `2g.20gb`: up to 3 per GPU - `3g.40gb`: up to 2 per GPU - `4g.40gb`: 1 per GPU - `7g.80gb`: the whole GPU, 1 per GPU Other parts use other names (a 40 GB A100 starts at `1g.5gb`), so check the table for the GPU you own. The sizes are fixed: a notebook that needs 15 GB has to be given a `1g.20gb` or `2g.20gb` instance, because MIG cannot carve 15 GB. ## Time-slicing through the device plugin - **`replicas`** under `sharing.timeSlicing.resources` makes the plugin hand out that many references to each GPU: eight GPUs with `replicas: 4` become 32 advertised resources. - **`renameByDefault`** (default `false`): when `true`, the resource is advertised as `nvidia.com/gpu.shared`, so a pod opts into sharing by name. When `false`, GPU Feature Discovery appends `-SHARED` to `nvidia.com/gpu.product` so a node selector can still tell shared nodes apart. - **`failRequestsGreaterThanOne`** (default `false` for backward compatibility, recommended `true`): a container asking for more than one shared replica fails with `UnexpectedAdmissionError` and must be deleted, fixed and redeployed. - CUDA gives an **equal share of time to every GPU process** across all pods, and each pod may run as many processes as it likes. A pod that requests two replicas does not get twice the compute, and nothing stops one notebook from allocating most of the 80 GB. - The config is a ConfigMap referenced from the ClusterPolicy's `spec.devicePlugin.config`; a `default` key applies it cluster-wide, and the node label `nvidia.com/device-plugin.config` selects a named config per node. The same method applies to all GPUs on a node. ## Choosing for the notebook pool Say four of the 32 nodes, 32 GPUs, serve notebooks: - **MIG, `all-1g.10gb`**: 32 x 7 = **224** isolated 10 GB instances. - **Time-slicing, `replicas: 4`**: 32 x 4 = **128** shared replicas, each able to touch all 80 GB. 1. Many users who do not trust each other, with memory needs that fit a profile: **MIG**. 2. GPUs without MIG, more than seven tenants per GPU, or one team's cooperating jobs: **time-slicing**. 3. Both: time-slicing also accepts the resources the mixed MIG strategy creates, so `nvidia.com/mig-1g.10gb` can carry `replicas` too. Instances stay isolated from each other; pods inside one instance share it. ## What a notebook pod asks for The request in the pod spec follows from the choice, which is why notebook images and templates must be written for one pool: - **MIG, single strategy**: `nvidia.com/gpu: 1`, plus a node selector on `nvidia.com/gpu.product`, which GPU Feature Discovery suffixes with the profile (for example `-MIG-1g.10gb`), so the pod lands on the sliced nodes. - **MIG, mixed strategy**: the profile is in the resource name, `nvidia.com/mig-1g.10gb: 1`. - **Time-slicing with `renameByDefault: true`**: `nvidia.com/gpu.shared: 1`, which no whole-GPU training pod will ever request by accident. - **Time-slicing with the default `false`**: `nvidia.com/gpu: 1` plus a node selector on the `-SHARED` product label; without it, the notebook could take a whole GPU on a training node. ## Operational caveats - The DCGM exporter cannot associate metrics with containers when time-slicing is on, which makes per-notebook accounting harder. - The operator does not watch the time-slicing ConfigMap for edits; updating it needs the documented reapply step. - Changing a node's MIG layout terminates the GPU pods on that node, so relayouts are maintenance events, not a per-notebook knob.
- Why turn on failRequestsGreaterThanOne in the device plugin's time-slicing config?It defaults to `false` for backward compatibility. Set to `true`, a container that asks for more than one shared replica fails with `UnexpectedAdmissionError`, which teaches users that a replica is an access ticket, not a fraction of compute. The pod then has to be deleted, fixed and redeployed.
- Can a MIG instance itself be time-sliced?Yes. Time-slicing supports `nvidia.com/gpu` and the resources the mixed MIG strategy creates, so `nvidia.com/mig-1g.10gb` can carry `replicas`. Instances stay isolated from each other, while the pods sharing one instance share its memory and its faults.
MIG is a building divided into flats, each with its own walls and meter; time-slicing is one shared flat where housemates take turns at a single kitchen. More people fit in the shared flat, but one housemate's mess stops everyone's dinner.
- Time-slicing gives every replica its own share of the GPU's memory.
- Requesting two time-sliced replicas doubles a pod's compute share.
- MIG can carve an 80 GB GPU into whatever size a pod requests.
- A notebook crashing on a time-sliced GPU affects only itself.
- MIG and time-slicing can never be used on the same GPU.
Assumes NVIDIA GPU Operator 26.7 with device plugin v0.20.1 and the H100 80 GB MIG profile table; other GPU models offer different profiles.