Kubernetes Is Turning Into the Resource Operating System for AI Clusters

Kubernetes v1.37 schedules devices, gangs, quotas and inference requests, not just Pods. A walkthrough of DRA, PodGroup, Kueue, JobSet, LWS and the inference gateway.

Cover illustration by the author: devices selected as a set, a gang admitted together, a queue behind a quota gate, leader-worker groups, and requests routed to model replicas

Sixty-four workers requested, forty placed, twenty-four stuck in Pending. The forty that got scheduled are holding expensive GPUs and computing nothing, because distributed training does not start until the last worker joins. Nobody is over quota, nothing crashed, and the cluster is doing exactly what a Pod-by-Pod scheduler was designed to do. That deadlock is what I was looking for when I read the Kubernetes v1.37 release notes in September 2026.

What I found was a control plane being rebuilt around it. Gang scheduling, hierarchical Pod groups, per-group device claims, scale-to-zero, request routing by KV cache state. The unit of scheduling is no longer the Pod.

Where the Pod Abstraction Stops Fitting

Kubernetes was built to run large numbers of independent stateless services. One Pod runs on its own, CPU and memory are the resources that matter, replicas are interchangeable, and round-robin is a reasonable way to spread traffic.

AI workloads violate every one of those assumptions.

A training job may claim dozens or thousands of GPUs and needs all workers to start together. GPUs with the same name can differ in memory size, interconnect topology and NUMA placement. A single inference replica may span several machines, with the model sharded across accelerators. Once you reach serving, request latency stops tracking connection count and starts tracking model state: KV cache, prefix cache, request length, which LoRA adapter is already resident, whether the endpoint is doing prefill or decode.

So the infrastructure problem is no longer "put this container on that machine". It is a different list:

Describing, selecting, sharing and configuring heterogeneous devices — GPUs, NPUs, RDMA NICs.
Getting a set of interdependent Pods resources at the same moment, instead of half of them idling while the rest wait.
Placing training processes according to rack, NUMA, NVLink and RoCE topology.
Queueing, quota, borrowing and fair preemption when several teams share expensive accelerators.
Managing a multi-role training job or a multi-host inference replica as one object.
Dispatching inference requests by model and cache state rather than connection count.
Scaling down when there is no traffic, and recovering quickly when something fails.

The last few Kubernetes releases have been working through that list.

DRA Turns "eight GPUs" into "this Kind of device"

The classic device plugin exposes a GPU as an integer extended resource:

resources: limits: nvidia.com/gpu: 8

That expresses "eight GPUs" and almost nothing else. It has no standard way to say the GPU needs at least 80 GB of memory, the GPU and the RDMA NIC must sit on the same NUMA node, that one model is preferred but a second is acceptable, that several Pods should share one device claim, that the GPU should be partitioned with MIG or vGPU, or a particular device is under maintenance and should be skipped.

Dynamic Resource Allocation (DRA) replaces the counter with a declarative model borrowed from dynamic storage provisioning. Platform teams describe device categories with a DeviceClass. Drivers publish devices and their attributes through ResourceSlice. Workloads state what they need through ResourceClaim or ResourceClaimTemplate. The scheduler then picks the node and the specific device in the same decision.

Underneath the new syntax, the scheduler now reasons about device-level facts:

CEL expressions filter devices by model, memory, topology and vendor attributes.
Device-specific configuration travels with the workload.
Containers, and multiple Pods, can reference and share a claim.
A DeviceClass can package messy hardware into platform products like "high performance" or "cost optimized".
CDI hands the allocated device, its device nodes and its runtime configuration to the container.

CDI Delivers the Device to the Container

The Container Device Interface is a standard device description consumed by container runtimes. DRA answers which device this workload gets; CDI answers how the container is set up to use it. CDI discovers nothing, counts no quota and makes no scheduling decision.

A GPU, NPU or DPU rarely works by mounting one /dev node. The container may also need driver libraries, management tools, environment variables, OCI hooks, extra device nodes and configuration matching a specific MIG or vGPU instance. That work used to be done by vendor-specific runtimes, startup hooks and scripts, which every container platform had to accommodate one vendor at a time. CDI writes it into a standardized spec and names devices uniformly:

nvidia.com/gpu=GPU-2f1c...
vendor.example/npu=card0

The whole path looks like this:

DRA driver discovers devices, publishes attributes via ResourceSlice
↓
kube-scheduler picks node and device from the ResourceClaim
↓
DRA driver prepares the device, returns CDI device names
↓
kubelet asks containerd or CRI-O for a container through CRI
↓
runtime reads the CDI spec and merges device nodes, driver
libraries, environment variables and hooks into the OCI spec
↓
runc / crun starts a container that can reach the device

DRA and CDI sit upstream and downstream of each other. DRA faces the control plane and owns declaration, selection, allocation and lifecycle. CDI faces the node runtime and performs standardized injection. Device plugins can also return CDI device names, so CDI is the delivery interface for both the old path and the new one.

Vendors generate a CDI spec, containerd and CRI-O execute it the same way, and nothing above needs to know which files a given accelerator wants mounted. Physical GPUs, MIG instances, vGPUs, NPUs and smart NICs all arrive through a similar route.

Device status and topology ride along. Status covers whether a device is healthy or under maintenance, and which device a ResourceClaim actually received. Topology covers the NUMA node, the PCIe position, and proximity to other devices such as NICs. The scheduler can use both to avoid failed hardware and to keep a GPU near its RDMA NIC. Richer topology — NVLink, NVSwitch, rack networks — still comes from DRA drivers, node labels, scheduler extensions or vendor stacks. Kubernetes core does not discover every hardware link on its own.

The DRA core reached GA in Kubernetes v1.34, and later releases filled in the production gaps. In v1.37, DRA support for extended resource requests went Stable: an extended resource name can be set directly on a DeviceClass, so a Pod asking for vendor.example/gpu: 1 gets matched to a device through DRA with no ResourceClaim of its own. Existing manifests keep working while the platform migrates off device plugins underneath them. The same release moved device taints and tolerations, the standard resource.kubernetes.io/numaNode attribute and per-device ResourceClaim status to Stable, and opened alpha work on derived attributes (CEL rules that pair devices whose vendors disagree on attribute names), device compatibility groups and node-allocatable CPU and memory managed through DRA.

Device management moved from counting units to modelling capabilities. For a cluster holding GPUs, NPUs, RDMA, DPUs and several virtualization formats at once, that is the prerequisite for being a single compute substrate.

Scheduling a Job, not Sixty-four Pods

Take the opening scenario again. Forty workers land, twenty-four go Pending, and the job holds GPUs while making no progress — a deadlock produced by a scheduler doing per-Pod arithmetic correctly.

The Workload and PodGroup APIs make a set of Pods one scheduling object. Gang scheduling binds the group atomically, or not at all, once the cluster can satisfy the declared member count.

In Kubernetes v1.37 the core APIs and gang scheduling graduated to Beta as scheduling.k8s.io/v1beta1, built on kube-scheduler's own filter, score and bind framework rather than a third-party batch scheduler bolted alongside it. Plain Kubernetes Jobs and higher-level controllers can share one native scheduling vocabulary. The feature gate ships disabled by default, so this is Beta you have to turn on: GenericWorkload on the API server, controller manager and scheduler. Which makes v1.37 the release to test it in, not the one to standardize on.

Two smaller changes landed with it. PodGroups are queued as a single object now, instead of Pod by Pod, so members share one queueing behavior; and minCount became mutable, which lets a controller resize a running gang instead of recreating it.

Preemption that Counts Whole Groups

Per-Pod preemption can evict a set of low-priority Pods and still fail to assemble what the training job needs. Workload-aware preemption treats the PodGroup as the unit, searches for victims across the cluster, and supports disrupting a whole group at once. It reached Beta in v1.37, folded into the same GenericWorkload gate, and default preemption now respects a PodGroup's disruptionMode instead of picking off its Pods individually.

Topology Enters the Placement Decision

Distributed training performance depends on the communication path between processes. The same number of GPUs scattered across racks or network domains can cut AllReduce throughput hard. Topology-aware scheduling lets a group declare rack or zone constraints; the scheduler generates candidate topology domains first, then checks whether the whole group fits inside one.

v1.37 extends this to multi-level constraints: a workload confined to one availability zone, with worker and driver groups each pinned to a rack inside that zone, resolved top-down. It remains Alpha behind TopologyAwareWorkloadScheduling. The direction is settled even if the API is not — free GPUs are not enough, the scheduler has to care how close they are.

One ResourceClaim for a Whole Group

DRAWorkloadResourceClaims graduated to Beta in v1.37. A group of Pods can share a ResourceClaim created or referenced by the PodGroup instead of each Pod maintaining its own, which lifts the old per-Pod reservation ceiling of 256 Pods per claim. For very large jobs sharing network devices or composite devices, that ceiling was a real limit.

CompositePodGroup and the Plumbing under it

Real AI jobs are not one flat pool of workers. A JobSet may carry a leader, workers and parameter servers; disaggregated inference carries prefill, decode and routing components.

v1.37 added the Alpha CompositePodGroup API, which describes a tree of groups and lets gang scheduling, preemption and topology constraints apply at each level. Alongside it came a set of controller integration building blocks and the workloadbuilder Go library, so JobSet, TrainJob, LeaderWorkerSet, RayJob and the native Job controller stop reimplementing the same translation logic. Validation there is deny-by-default: a controller lists the policies it supports, and anything else is rejected.

The native batch/v1 Job picked up an Alpha spec.scheduling field carrying exactly those blocks — gang policy, topology constraints, disruption mode, shared resource claims — instead of having the controller infer intent from the Job's shape. AI job scheduling is moving from bolt-on into the standard workload API.

Kueue Decides Who Gets to Start

Gang scheduling answers how a group of Pods lands on nodes. It says nothing about which team's job runs first, or how many GPUs each team may hold. That is Kueue's question — a Kubernetes SIG Scheduling project for queueing, quota and workload admission.

Kueue does not replace kube-scheduler. It decides, before Pods are created, whether a Job is allowed to start consuming resources at all.

High-level Kueue operation: jobs are admitted against quota and flavors before the Job controller ever creates Pods. Source: Kueue documentation, the Kubernetes Authors, CC BY 4.0

What it provides:

Priority queueing with StrictFIFO and BestEffortFIFO strategies.
Quota per team, namespace and resource type.
ResourceFlavor to distinguish H100 from A100, one NPU from another, spot from on-demand.
Cohorts for borrowing idle quota between queues, with fair sharing.
Preemption of lower-priority work when policy allows it.
Partial admission, dynamic reclaim, and waiting on cluster autoscaler capacity through provisioningRequest.
Topology-aware admission, so quota is not granted for a placement that cannot form.
MultiKueue for finding capacity across clusters and dispatching there.
Integrations covering batch Jobs, Kubeflow training jobs, RayJob, JobSet, LeaderWorkerSet, plain Pods, Deployments and StatefulSets.

In a full platform the two layers split cleanly: Kueue decides when a job may start and against which quota, kube-scheduler decides which nodes the Pods or PodGroups land on. Kueue's own all-or-nothing admission is timeout-based, which is why the v1.37 roadmap talks about Kueue eventually leaning on workload-aware scheduling as its gang engine rather than keeping two implementations.

JobSet Models Multi-role Training

A single Job suits a set of identical Pods. Many training jobs are not that: launcher, leader, workers, parameter servers, data processing nodes. JobSet, from SIG Apps, gives those an API where one job is composed of several Jobs.

A JobSet composed of replicated Jobs, with an automatically managed headless Service for Pod-to-Pod communication. Source: Introducing JobSet, the Kubernetes Authors, CC BY 4.0

What it buys you:

Multiple templates for different roles and resource shapes.
Automatic headless Service management, giving training processes stable hostnames and discovery.
Startup ordering: driver first for Ray or Spark, workers ready before the driver for MPI.
Success, failure and restart policies defined for the whole set.
Recreation of the entire JobSet after a child failure, so the workload resumes from its last checkpoint.
Exclusive placement of child Jobs onto separate racks or topology domains.
Kueue integration, so a complete multi-role job is queued and charged as one unit.

JobSet owns none of the PyTorch, JAX or TensorFlow logic. It turns the multi-role processes, network identities and lifecycles those frameworks assume into objects Kubernetes can manage.

LeaderWorkerSet and DisaggregatedSet for Multi-host Inference

Deployment assumes replicas are independent and homogeneous. A very large model replica may span machines: a leader coordinating, workers each holding a shard. To Kubernetes, that group should look like one super Pod.

LeaderWorkerSet (LWS) is built for it:

A leader plus its workers form one unit of replication and lifecycle.
Each Pod in the group gets a stable index identity from 0 to n-1.
Parallel creation, opt-in gang scheduling and topology-aware placement.
Scaling, rolling updates and failure recreation happen at group granularity.
The whole group enters or leaves service together, so you never serve half-upgraded shards.

As serving moved from plain tensor parallelism toward prefill/decode disaggregation, the project added DisaggregatedSet, which runs each role as its own child LWS. It coordinates rollouts across 2 to 10 roles in lockstep while preserving capacity ratios, manages their headless services, and handles coordinated drain and restart. Both APIs are being co-designed with llm-d, the CNCF sandbox inference stack.

The object being managed is no longer a stateless replica. It is a distributed model replica with internal topology, role relationships and a shared lifecycle.

Routing Requests by Model State

Round-robin, random and least-connections balancing know nothing about what an inference request costs.

Two Pods can report identical connection counts while one already holds the prompt prefix in cache and the other does not; sending the request to the first can cut time-to-first-token sharply. A node with the right LoRA adapter already resident avoids reloading weights.

Gateway API Inference Extension is the Kubernetes project for this. It adds InferencePool and an endpoint picker to Gateway API, so the router chooses endpoints using metrics and capabilities reported by model servers.

Request flow through an inference gateway: body-based routing selects the pool by model name, then endpoint selection picks the least-loaded replica holding the right adapter. Source: Introducing Gateway API Inference Extension, the Kubernetes Authors, CC BY 4.0

The problems it targets:

Routing by model name rather than URL path alone.
Awareness of prefix cache, KV cache, LoRA adapters and endpoint load.
Different service priorities for interactive chat versus batch summarization.
Traffic splitting and progressive rollout across model versions.
Pluggable endpoint-selection algorithms optimizing for cost, latency or throughput.
Lower inference latency and better GPU utilization from smarter dispatch.

The project's own benchmark ran vLLM v1 on H100 80 GB pods with 10 Llama2 replicas, driving ShareGPT traffic from 100 to 1000 QPS against a standard Kubernetes Service as the baseline.

Throughput stays comparable while p90 per-output-token latency and p90 end-to-end latency drop once traffic passes roughly 400 to 500 QPS. Source: Introducing Gateway API Inference Extension, the Kubernetes Authors, CC BY 4.0

One thing to know before you build on it: the Endpoint Picker and the InferenceObjective API have moved out of kubernetes-sigs/gateway-api-inference-extension into llm-d/llm-d-router. The SIG repository keeps the InferencePool API, a lightweight reference EPP and the conformance tests.

Scheduling now covers both halves of the problem. Offline placement decides where a model runs; online routing decides which replica serves each request.

Elasticity, Recovery, and a Control Plane that Keeps up

Scaling to Zero

HPA scale to zero reached Beta in v1.37 and is enabled by default. Workloads driven by object or external metrics can drop to zero Pods when idle and come back when the queue fills — valuable for inference services sitting on expensive GPUs with peaky traffic. CPU and memory metrics are excluded, since those depend on running Pods, and the HPA records a ScaledToZero condition so the controller can tell its own zero from a manual one.

Scaling to zero does not solve slow weight loading, GPU initialization or cold start. Real platforms still need model caching, image distribution, predictive scaling or a warm replica held back.

Pod-level Checkpoint and Restore

v1.37 introduced Alpha Pod-level checkpoint and restore, extending CRI with CheckpointPod and RestorePod RPCs so kubelet and a compatible runtime can save and restore a whole Pod. That gives long training runs a new foundation for failure recovery, node maintenance and migration. Whether GPU, network and distributed training state survive the round trip depends on the runtime, the driver and the framework above, so treat it as a building block rather than a replacement for framework checkpointing.

Control Plane Throughput

AI clusters carry a lot of nodes, Pods, device objects and custom controllers, and recent releases have been grinding on that. v1.37 shipped etcd RangeStream at Beta, on by default, so the watch cache stops assembling entire lists in memory (it needs etcd 3.7+ and falls back automatically on older versions). ConcurrentWatchObjectDecode, in Beta since v1.31, is now on by default: decoding watch events across a bounded pool of 10 goroutines cut cache initialization by about 40% over 150k Pods in the project's benchmarks, and roughly 55% combined with RangeStream. Resilient watch cache initialization went Stable and is locked on.

None of that is labelled AI. It decides whether a ten-thousand-GPU cluster and its serving platform stay up.

Five Layers that Compose

Layer	Capabilities	Question answered
Device	DRA, CDI, device health, NUMA and interconnect topology	Which device, is it usable, and how is it selected, combined, configured and delivered
Job scheduling	Workload, PodGroup, gang scheduling, group preemption	How a set of interdependent Pods gets resources at the same moment
Queue and quota	Kueue	How tenants queue, share fairly and find capacity across clusters
Workload orchestration	JobSet, LeaderWorkerSet, DisaggregatedSet	How multi-role training and multi-host, disaggregated inference are managed
Online traffic	Gateway API Inference Extension	Which model instance serves this request

They compose rather than compete. A training platform might claim GPUs and RDMA through DRA, manage team quota with Kueue, describe the multi-role job with JobSet, and let PodGroup plus kube-scheduler handle group and topology placement. A serving platform might allocate devices with DRA, deploy replicas with LWS or DisaggregatedSet, scale with HPA, and dispatch requests through the inference gateway based on cache and model state.

What Kubernetes now Answers

Kubernetes will not replace CUDA, PyTorch, vLLM, SGLang or Ray, and it runs no training or inference algorithm itself. Its job is organizing compute, network, storage, queues, jobs and services into declarative, extensible, portable infrastructure.

The question it used to answer best was: which node should this Pod run on?

It is learning to answer more:

What kind of GPU, NIC and topology does this job need?
Can this set of workers start together?
Which team is entitled to this capacity right now?
How should a multi-host model replica be upgraded and recovered?
Which model instance should serve this request?

Kubernetes stays general-purpose infrastructure while learning the resource semantics of AI workloads. That is the shift worth tracking: from container orchestration platform toward the resource operating system and control plane of an AI data center.