Sessions, Not Requests, Are Becoming the Real Unit of AI Compute
Session-based compute is an AI infrastructure pattern where platforms schedule, isolate, and bill long-running agent sessions as the core unit of work instead of treating each interaction as an independent, stateless request in a generic load-balancing pool. That shift is the most important architectural story in cloud AI right now: AWS, Microsoft, Google, and Anthropic have all rebuilt their AI agent runtimes around sessions as the operational unit of enterprise AI infrastructure. This is not a cosmetic API tweak; it is a quiet admission that stateful AI workloads break the assumptions that powered twenty years of web-scale cloud design. If you are building production AI agents today and you are still thinking in terms of requests, you are designing against the grain of where the major platforms are going.
Why Stateless Load Balancing No Longer Works for Stateful AI Workloads
Traditional web scaling worked because two assumptions held: requests did not depend on one another, and any backend could serve any request. That made it safe to put NGINX or HAProxy in front of a worker pool, keep state in Redis or a database, and treat sticky sessions as a rare exception. Stateful AI workloads destroy both assumptions. Enterprise agents are long-running, stateful, tool-using processes that often run code influenced by user input. When an agent processes a refund, calls tools, and waits for a model, the next turn needs the same conversational state; if a different replica without that context picks it up, the workflow breaks. The load balancer stops being a traffic shuffler and starts behaving like a session router, which means the session—not the request—must be the thing the platform understands, isolates, and manages.
Four Cloud Approaches to Session Isolation, One Architectural Direction
Over the past few months, AWS, Microsoft, Google, and Anthropic each shipped agent runtime updates that converge on the same idea: per-session execution as the baseline for AI agent runtimes. Anthropic’s Managed Agents explicitly virtualize the agent into a session, a harness, and a sandbox. AWS routes each session into its own microVM and documents a lifecycle where an AgentCore session is Active while processing, Idle when provisioned but waiting, and Terminated after a 15-minute idle timeout or an 8-hour maximum lifetime. Microsoft rebuilt its Foundry-hosted agents around per-session isolation and assigns each agent a dedicated Microsoft Entra identity, with sessions enforcing a 15-minute idle timeout against a 30-day maximum lifetime. Google isolates agent code execution inside dedicated sandboxes. The execution substrate differs, but the architectural direction is identical: session-aware execution is becoming a baseline requirement for production-grade agents.
Security Incidents and Kubernetes-Driven AI Push the Shift to Sessions
The move to session-based compute is not fashion; it is a response to hard failures and new infrastructure realities. An agent can execute model-generated, user-shaped code, so the backend must be treated as a security boundary, not a generic worker. A shared kernel without strong tenant isolation is no longer acceptable. When Asana disclosed a flaw in its MCP server in June 2025, the server had been live since May 1, about five weeks; a tenant-isolation check failed, and around 1,000 organizations saw project data from other customers. The incident showed that user identity, agent context, and session state must be consistently bound. At the same time, cloud providers are redesigning their stacks for AI: Microsoft’s latest Azure Kubernetes Service enhancements—AKS on Bare Metal, Fleet Manager, Anyscale on Azure, AI Runway, and KAITO—aim to make Kubernetes a first-class platform for AI training, inference, and large-scale applications.
What Session-Based Compute Means for Cost, Performance, and Design
Once the session is the unit of compute, everything from cost to performance changes. When billing is tied to active sessions, concurrency, idle time, and agent sizing become the cost drivers rather than request volume. The better mental model is closer to a virtual actor runtime: an addressable identity is instantiated on demand, kept active while needed, and deactivated when idle, with one live instance per key. That model favors long-running, stateful AI agents over stateless API calls, and it rewards designs that minimize idle capacity while keeping context local for performance. It also aligns with a broader cloud infrastructure redesign in which providers abstract Kubernetes operations so teams can focus on AI workloads, not cluster plumbing. The session becoming the operational unit of enterprise AI infrastructure is the clearest signal that agents—not simple prompts—are what cloud platforms expect you to run.
