From Platforms to AI Factories: Architectural Takeaways from KubeCon India 2026

If you are a DevOps engineer or platform architect, you already know that building Internal Developer Platforms (IDPs) has been the dominant theme for the last few years. But walking the floor at KubeCon + CloudNativeCon India 2026 at the Jio World Convention Centre in Mumbai, it became immediately clear that the goalposts have moved.

The industry is rapidly shifting from building standard stateless developer platforms to architecting true AI Factories. The new hard problems aren't just CI/CD or basic cluster autoscaling; they are GPU orchestration, stateful agent deployments, and distributed inferencing at scale.

Here is a technical breakdown of the most critical infrastructure shifts I took away from the sessions and keynotes.

Escaping the "One Pod, One GPU" Trap

In his keynote, Saiyam Pathak hit on the exact bottleneck platform teams are facing: how do you safely share GPU infrastructure across multiple AI teams without stepping on each other?

Historically, Kubernetes has been blunt with GPUs, allocating them in whole integers. Even if a workload only requires 10% of a GPU's compute, the entire device gets locked to that pod. That is financially unsustainable for multi-tenant SaaS.

The CNCF ecosystem is solving this through two major shifts:

• HAMi (Heterogeneous AI Computing Virtualization): This allows us to virtualize a single physical GPU into multiple slices, each with its own isolated memory budget.

• DRA (Dynamic Resource Allocation): DRA fundamentally evolves the K8s resource model so the scheduler understands GPUs as rich, dynamic devices rather than opaque numbers.

The live demo—a MacBook connected to an NVIDIA DGX Spark utilizing a single Blackwell GPU to run two open-source LLMs simultaneously for different teams—proved that Kubernetes is finally ready for true multi-tenant AI workloads.

Distributed Inferencing with llm-d

Running AI workloads on Kubernetes has moved past basic cost tuning. The new frontier is handling complex LLM inferencing efficiently across a cluster.

Ravindra Patil (Red Hat) delivered an excellent session on llm-d, a high-performance distributed inference framework created by Red Hat, Google Cloud, IBM, CoreWeave, and NVIDIA. Recently accepted as a CNCF sandbox project, llm-d tackles the inefficiencies of standard load balancers (which are blind to GPU utilization and KV cache state).

It optimizes inferencing via two critical mechanisms:

• Prefill/Decode Disaggregation: The compute-heavy "prefill" phase and the memory-bandwidth-heavy "decode" phase are separated, allowing them to scale independently across specialized workers.

• Intelligent Token-Aware Routing: Instead of round-robin routing, llm-d routes requests based on real-time prefix-cache awareness, meaning requests with similar prompts hit the nodes where the KV cache already exists.

The "Paved Road" for AI Agents

AI agents are moving from local Python scripts into production, but agentic workloads don’t fit neatly into traditional stateless 12-factor app templates. They require long-running sessions, database access, and tool execution, which often leads to operational inconsistency.

Two sessions stood out for solving this:

• Self-Contained Data Agents (Cisco): Shuva Jyoti Kar showcased a blueprint for standardizing agents by packaging the runtime, tool contracts, and the context database behind a clear containerized service boundary. By leveraging Knative, platform teams can offer request-driven deployments with scale-to-zero capabilities and robust session management without sacrificing the native K8s experience.

• gRPC for the Model Context Protocol (Google): The Model Context Protocol (MCP) standardizes how agents interact with data sources. Pawan Bhardwaj proposed using gRPC as the native transport for MCP. Because gRPC relies on Protobuf, it enforces strict API contracts between agents and tools. More importantly, running MCP over gRPC allows platform teams to tap directly into proxy-less service mesh features, inheriting mTLS, advanced load balancing, and auth frameworks without building them from scratch.

Securing Agentic Workloads

As AI agents gain the ability to execute tools and modify state, traditional IAM and API gateway security models fall short.

Rahul Jadhav (AccuKnox) delivered a standout security session on building SPIFFE & OpenFGA Based Identity/Authz for Agentic AI. He demonstrated how to move beyond static API keys by provisioning cryptographic, short-lived workload identities via SPIFFE. Coupling this with OpenFGA allows platforms to evaluate fine-grained authorization dynamically. If an agent tries to access a restricted bucket or execute a dangerous tool, the mesh denies the request based on real-time relationship-based access control (ReBAC) rather than flat RBAC roles.

Day 2 Operations, Scale, and FinOps

While AI took the spotlight, the fundamentals of high-scale engineering and day 2 operations remain the lifeblood of the cloud-native ecosystem.

• Observability at Scale: Aditi Gupta (JioHotstar), Madhu Patel (Adobe), and Sandeep Kanabar (Gen) delivered an incredible session titled Who Watches the Watchers?. They detailed the transition from closed observability systems to open control planes, sharing real-world telemetry architectures for streaming environments serving millions of concurrent users.

• The Upgrade Playbook: Yug Gupta (Walmart Global Tech) provided a masterclass in surviving technical debt with The Leapfrog Upgrade Playbook. Upgrading a K8s cluster when you are multiple minor versions behind is universally dreaded. His operational playbooks for skipping versions while maintaining workload resiliency are essential reading for any infrastructure team.

• Automated FinOps: Avinash Gupta (PhysicsWallah) and Gaurang Singh (Cast AI) demonstrated hyperscale FinOps in action. By implementing automated rightsizing and dynamic bin-packing, they successfully reduced Kubernetes CPU requests by 35%—handling massive traffic spikes during live EdTech classes without dropping a single connection.

Final Thoughts

KubeCon India 2026 made one thing abundantly clear: Kubernetes has won the orchestration war, and it is now absorbing the AI workload ecosystem. If you are a platform engineer, your next 12 months will likely be spent figuring out GPU slicing, deploying distributed inferencing gateways, and securing stateful agents with SPIFFE.