Consider a platform team managing a shared GPU cluster with a mix of B200s, H100s, and recently added B300s. Every Monday morning, the on-call engineer finds a queue of pending jobs from the weekend. Training workloads are stuck because they landed on H100s and triggered Out-Of-Memory (OOM) errors. Inference jobs sit idle because the small MIG (Multi-Instance GPU) slices are exhausted, even though larger slices sit empty right next to them.Their fix? A 200-line Bash script running every 30 minutes to reconfigure MIG profiles, reschedule stuck jobs, and send a Slack alert when it succeeds, or a PagerDuty alert when it fails.The root of the problemHere is what was actually broken: Kubernetes treated every GPU as an identical unit. The resource limit nvidia.com/gpu: 1 was the extent of its awareness. The scheduler had no idea if it was handing a pod a 192GB B200 or an 80GB H100. A training job requiring 150GB of VRAM would land on an H100 and immediately OOM, while B200 nodes sat completely idle nearby.“Kubernetes treated every GPU as an identical unit.”The industry’s accepted “fix” relied heavily on node labels, taints, tolerations, and separate node pools per GPU type. Every workload manifest hardcoded hardware assumptions. Adding a single new GPU generation meant updating 40 different Helm charts.The MIG illusionMIG made this worse. MIG slices a single GPU into smaller, isolated partitions, each with dedicated memory and compute. Instead of one inference job monopolizing a B200, you can run seven smaller jobs on the same card.In theory, this sounds great. But when you enable MIG in Kubernetes, each profile becomes a separate, rigid resource type (e.g., nvidia.com/mig-1g.10gb, nvidia.com/mig-3g.40gb). There is no fallback logic. You cannot instruct a job to “try a small slice first, and use a large one if nothing else is free.” When small slices run out, jobs sit pending, while large slices go to waste.“When small slices run out, jobs sit pending, while large slices go to waste.”This inefficiency was accepted as the cost of running GPU workloads on Kubernetes. Then, Kubernetes 1.34 shipped.Dynamic Resource Allocation (DRA)Kubernetes 1.34 introduced Dynamic Resource Allocation (DRA), fundamentally changing the scheduling model. GPU drivers now publish structured data. Instead of requesting a generic nvidia.com/gpu: 1, workloads can express explicit intent using Common Expression Language (CEL):“Give me an H100 or better with at least 40GB of memory.”“Give me a MIG slice: small if available, medium if not, or a full GPU if necessary.”“Give me four GPUs connected via NVLink.”Example 1: Hardware and memory requirements“Give me an H100 or better with at least 40GB memory.”apiVersion: resource.k8s.io/v1kind: ResourceClaimTemplatemetadata: name: h100-or-better-40gbspec: spec: devices: requests: - name: gpu deviceClassName: gpu.nvidia.com count: 1 selectors: - cel: # Attribute names are illustrative. # Your NVIDIA DRA driver must expose these fields. expression: > device.attributes["gpu.nvidia.com"].memory >= quantity("40Gi") && device.attributes["gpu.nvidia.com"].generation in ["H100", "B200", "B300"]---apiVersion: batch/v1kind: Jobmetadata: name: training-job-h100-or-betterspec: template: spec: restartPolicy: Never resourceClaims: - name: gpu source: resourceClaimTemplateName: h100-or-better-40gb containers: - name: trainer image: nvcr.io/nvidia/pytorch:24.12-py3 command: ["python", "train.py"] resources: claims: - name: gpuExample 2: Flexible MIG fallback“Give me a MIG slice: small if available, medium if not, or full GPU if needed.”apiVersion: resource.k8s.io/v1kind: ResourceClaimTemplatemetadata: name: mig-prefer-small-then-medium-then-fullspec: spec: devices: requests: - name: gpu deviceClassName: gpu.nvidia.com count: 1 selectors: - cel: # Prefer any acceptable MIG profile or full GPU expression: > device.attributes["gpu.nvidia.com"].profile in [ "mig-1g.10gb", "mig-2g.20gb", "mig-3g.40gb", "full-gpu" ]---apiVersion: apps/v1kind: Deploymentmetadata: name: inference-service-flexible-gpuspec: replicas: 2 selector: matchLabels: app: inference-service template: metadata: labels: app: inference-service spec: resourceClaims: - name: gpu source: resourceClaimTemplateName: mig-prefer-small-then-medium-then-full containers: - name: inference image: nvcr.io/nvidia/tritonserver:24.12-py3 args: ["tritonserver", "--model-repository=/models"] resources: claims: - name: gpuExample 3: Topology constraints“Give me 4 GPUs that are NVLink-connected.”apiVersion: resource.k8s.io/v1kind: ResourceClaimTemplatemetadata: name: four-nvlink-connected-gpusspec: spec: devices: requests: - name: gpus deviceClassName: gpu.nvidia.com count: 4 selectors: - cel: # Attribute names are illustrative. # Some NVIDIA DRA setups may model this through ComputeDomains. expression: > device.attributes["gpu.nvidia.com"].fabric == "nvlink" constraints: - requests: ["gpus"] matchAttribute: "gpu.nvidia.com/nvlinkDomain"---apiVersion: batch/v1kind: Jobmetadata: name: distributed-training-nvlinkspec: template: spec: restartPolicy: Never resourceClaims: - name: gpus source: resourceClaimTemplateName: four-nvlink-connected-gpus containers: - name: trainer image: nvcr.io/nvidia/pytorch:24.12-py3 command: - torchrun - --nproc_per_node=4 - train.py resources: claims: - name: gpusThe engineering takeawayThis architecture requires one manifest. It eliminates fragile node selectors and the need to duplicate job definitions for every new hardware generation. As GPU clusters become increasingly heterogeneous, mixing H100s, B200s, B300s, and whatever silicon drops next, the old integer-based scheduling model breaks down. DRA represents Kubernetes finally maturing to support the nuanced realities of production AI workloads.The post Say goodbye to K8s GPU pain: How DRA changes everything appeared first on The New Stack.