How Kubernetes Became the Backbone of AI Infrastructure — and Why NVIDIA and AMD Are Betting on It
A decade ago, Kubernetes was best known for orchestrating stateless web services: scaling pods behind a load balancer, rolling out microservices, keeping e-commerce sites alive during traffic spikes. Today, the same platform is quietly running some of the largest AI training and inference clusters on the planet. That shift didn't happen by accident, and it hasn't gone unnoticed by the companies that make the chips those clusters depend on. NVIDIA and AMD are now among the most active contributors to the Kubernetes ecosystem, open-sourcing schedulers, drivers, and operators that used to be closely-guarded proprietary tooling.
This post looks at how Kubernetes earned its place as the default control plane for AI workloads, and why GPU vendors have decided that open source — not proprietary lock-in — is the smarter long-term bet.
Why AI workloads needed a new kind of platform
Training and serving large models is a fundamentally different problem than running a typical web app. A few things make it hard:
GPUs are expensive and scarce: Idle accelerator time is money burned, so scheduling has to be tight and fair across teams.
Jobs are bursty and interdependent: Distributed training needs "gang scheduling" — all pods in a job must start together or not at all — while inference workloads need to scale instantly with traffic.
Hardware is heterogeneous: A single cluster might mix H100s, MI300Xs, and older cards, each with different memory, topology, and interconnect characteristics.
Serving has its own traffic patterns: Routing a request to the right model replica depends on things a normal load balancer doesn't understand — which LoRA adapter is loaded, how full the KV cache is, or which pod already has the right weights warmed up.
Kubernetes wasn't originally built for any of this. But its extensibility — Custom Resource Definitions, operators, a pluggable scheduler — meant the community could bolt on GPU-awareness without forking the project. That extensibility is arguably the single biggest reason Kubernetes won: instead of one vendor building a closed AI platform, hundreds of contributors could build specialized tooling on a shared foundation, and users could mix and match.
The ecosystem has matured fast
A few milestones show how quickly "Kubernetes for AI" has gone from a patchwork of workarounds to a genuinely purpose-built platform:
Dynamic Resource Allocation (DRA) reached general availability in core Kubernetes, replacing the old device-plugin model that had been essentially unchanged since 2017. Under the old model, you asked for nvidia.com/gpu: 1 and hoped for the best — the scheduler had no idea which physical GPU you got, what memory it had, or how it was wired to other GPUs. DRA introduces declarative ResourceClaims and CEL-based filtering, enabling fine-grained, topology-aware GPU allocation — including splitting a single GPU into fractional slices for multiple workloads with hardware-level isolation.
The Gateway API Inference Extension reached GA as a Kubernetes-native way to route traffic specifically for model serving — directing requests based on model name, LoRA adapter, or endpoint health, rather than generic round-robin load balancing. It's already seen adoption from vendors including NVIDIA and Alibaba Cloud.
Kueue and Volcano now handle admission control and gang scheduling for batch AI jobs, letting platform teams enforce quotas and fair-share policies across teams competing for the same GPU pool.
Kubernetes' WG Serving, a dedicated working group formed specifically to mature the AI inference stack, recently wrapped up its work — not because the job is done, but because the patterns it developed (Kueue, inference gateway, DRA, leader-worker sets) were successful enough to fold into standard SIGs and an official Kubernetes AI Conformance program.
llm-d, a project for disaggregated LLM serving that splits the "prefill" and "decode" phases of inference across different pods, was donated to the CNCF by IBM, Red Hat, and Google Cloud — a sign that even direct competitors see more value in a shared open standard than in separate proprietary stacks.
None of this happened in a vacuum. It happened because the two biggest GPU vendors on Earth decided to show up and contribute code, not just certification badges.
NVIDIA: from proprietary platform to open contributor
NVIDIA's Kubernetes strategy used to look a lot like its broader software strategy: build a complete, polished platform (like the Run:ai scheduler it acquired) and sell access to it. That's changed noticeably over the past two years.
The clearest example is the KAI Scheduler. Originally a core, proprietary component of the Run:ai platform NVIDIA acquired, it was released under the Apache 2.0 license as a fully open-source Kubernetes-native GPU scheduler. KAI handles the messy realities of multi-tenant GPU clusters: fluctuating demand, GPU fragmentation, priority queues, gang scheduling, and preemption — the kind of scheduling logic that used to require an expensive commercial platform. It's built on the community's existing kube-batch foundation rather than a walled garden, and it's now integrated with projects like KubeRay for Ray-based workloads.
Even more significant: NVIDIA donated its Dynamic Resource Allocation driver to the CNCF itself, rather than keeping it as a proprietary add-on tied to its own tooling. That's a meaningful signal — DRA is the mechanism that determines how every workload on a cluster gets access to a GPU, and handing that donation to a neutral foundation means NVIDIA is betting on Kubernetes becoming the common interface for accelerators generally, not just a distribution channel for NVIDIA-specific software.
NVIDIA has also pushed Dynamo, its open inference-serving framework for disaggregated LLM deployment, and has been an active participant in the Gateway API Inference Extension effort — the routing layer that decides which GPU-backed pod actually serves a given request.
AMD: building the on-ramp for ROCm on Kubernetes
AMD's motivation is a little different but points the same direction. For years, one of the biggest practical barriers to using AMD GPUs for AI workloads wasn't the hardware — it was the tooling gap around CUDA's ecosystem. Kubernetes has become AMD's chosen battleground for closing that gap in the open, rather than trying to build a closed competitor to NVIDIA's stack.
AMD maintains an open-source Kubernetes device plugin for ROCm, which registers AMD GPUs as schedulable resources in a cluster, and has layered a full AMD GPU Operator on top of it. The operator automates what used to be manual, node-by-node toil: installing the ROCm driver stack, deploying the device plugin and node labeller, and exposing Prometheus-compatible GPU telemetry — all on vanilla upstream Kubernetes as well as Red Hat OpenShift. Later releases added support for AMD's Dynamic Resource Allocation driver as an alternative to the classic device-plugin model, keeping pace with where upstream Kubernetes itself is heading, plus node-problem-detector integration and automated recovery for unhealthy GPU nodes.
Crucially, none of this lives behind a paywall or a proprietary control plane — it's all developed in the open on GitHub, versioned alongside ROCm releases, and designed to plug into the same CNCF tooling (Helm, cert-manager, standard Kubernetes APIs) that any other operator uses. For AMD, being a first-class, well-documented citizen of the open Kubernetes ecosystem is a more credible path to developer trust than trying to win people over with a closed alternative.
Why open source, specifically?
It would be easy to read this as generic "vendors love open source" marketing, but there's a sharper strategic logic underneath it.
Neutral ground lowers switching costs — for everyone. If GPU scheduling, device plugins, and inference routing all speak the same Kubernetes-native APIs, a platform team can mix NVIDIA and AMD hardware in one cluster, or migrate workloads between them, without rewriting their orchestration layer. Projects like HAMi, a CNCF Sandbox project for GPU resource management, explicitly aim to work across NVIDIA, AMD, Huawei, and Cambricon accelerators — something no single vendor's proprietary stack would ever offer.
Standards attract the ecosystem that sells the hardware. Neither NVIDIA nor AMD makes money directly from a Kubernetes scheduler. They make money when more workloads run on their silicon. Contributing the scheduling and orchestration layer to a shared, trusted open-source commons — rather than a vendor-controlled platform — removes friction for the enterprises, cloud providers, and startups deciding where to point their next GPU order.
Community-vetted code earns trust that marketing can't. A GPU driver or scheduler that handles multi-tenant resource isolation is security- and reliability-critical infrastructure. Having that code open, forkable, and reviewable by the CNCF community — rather than a black box — matters a great deal to platform teams responsible for keeping expensive clusters both efficient and safe.
It follows where Kubernetes governance was already headed. The CNCF's neutral governance model was built precisely so that no single vendor could capture core infrastructure. By donating pieces like the DRA driver into that model instead of keeping them proprietary, NVIDIA and AMD are aligning themselves with — rather than fighting — the norms that made Kubernetes trustworthy enough for AI workloads to land on it in the first place.
What this means going forward
The direction of travel is fairly clear. Kubernetes is no longer just "one option" for running AI infrastructure — recent CNCF survey data puts Kubernetes production usage among container users above 80%, and that number only grows as AI workloads join the rest of the stack rather than living in bespoke silos. The layers that used to require expensive, vendor-specific platforms — fair GPU scheduling, topology-aware allocation, inference-aware routing — are steadily becoming standard, open, community-governed parts of Kubernetes itself.
For NVIDIA and AMD, that's not a threat to their business models; it's an on-ramp. Every open-sourced scheduler, driver, and operator makes it a little easier for the next AI team to stand up a GPU cluster without reinventing infrastructure from scratch — and a little more likely that when they do, both vendors' hardware is a first-class citizen rather than an afterthought. In an industry defined by compute scarcity, that kind of ecosystem investment may end up mattering as much as the silicon itself.