Sessions Become the New Unit of Compute
Session-based compute is an AI runtime architecture in which cloud platforms schedule, meter, and secure long-running agent sessions as the primary unit of work instead of treating each request as an independent, stateless transaction.
Cloud agent infrastructure is quietly undergoing its biggest shift since the birth of microservices: the move from request-level routing to session-aware execution. AWS, Microsoft, Google, and Anthropic have each rebuilt their agent runtimes around the idea that the session—not the HTTP request—is now the real unit of compute. The trigger is not fashion; it is the hard edge of production workloads. Enterprise agents are long-running, stateful, tool-using processes that run code shaped by user input, and that combination breaks the assumptions behind classic load balancers. In this world, treating sessions as first-class objects is no longer a performance tweak. It is the baseline for correctness, security, and predictable cost.
The industry has converged on one conclusion: session-aware execution is becoming a baseline requirement for production-grade agents. But while the giants agree on the new unit of abstraction, they are taking very different paths on how tightly and how expensively to isolate it. That disagreement will decide which platforms feel safe enough for untrusted code and affordable enough for everyday automation.
Why Stateless Load Balancing Failed the Agent Era
For two decades, the dominant AI runtime architecture for web backends rested on a simple model: front-end load balancers like NGINX spray requests across a pool of workers, while state lives elsewhere. Any backend can serve any request because the application carefully avoids tying identity or context to a specific process. Sticky sessions existed, but they were treated as the odd exception, not the rule.
Agents blow up both assumptions. They must preserve conversational state over many turns and enforce a strong trust boundary around untrusted, model-generated code. Routing alone can only solve the first problem. Even with consistent hashing and session affinity, the load balancer morphs from traffic splitter into a session router, keeping a warm environment alive per conversation. That is no longer a best-effort optimization; in agents, session binding becomes a correctness and security requirement.
The consequences of getting that binding wrong are already visible. When Asana disclosed a flaw in its MCP server in June 2025, a tenant-isolation check failed and roughly 1,000 organizations saw project data from other customers. No external attacker was needed; misbound identity, agent context, and cached responses were enough. The lesson is blunt: once sessions become the unit that ties together user, agent, and state, they also become the unit that the platform must schedule and harden.
How the Giants Are Rebuilding Cloud Agent Infrastructure
Over the past few months, AWS, Microsoft, Google, and Anthropic all shipped agent runtime updates that share a striking pattern: each platform now treats the agent session as the operational unit of enterprise AI infrastructure. The common design is not a new model feature; it is a control plane for state, identity, isolation, and lifecycle, where the scheduled object is a session rather than a request.
Microsoft rebuilt its Foundry-hosted agents around per-session isolation, assigning each agent its own identity and session lifecycle. That philosophy extends down the stack. At its Build event, Microsoft unveiled AKS on Bare Metal, Azure Kubernetes Fleet Manager, a managed Ray service via Anyscale, and AI-focused tooling like AI Runway and the Kubernetes AI Toolchain Operator. Together, they aim to make Kubernetes the backbone for AI training, inference, and large-scale applications while exposing familiar primitives to platform engineers.
Anthropic’s Managed Agents split the world into session, harness, and sandbox, reinforcing that the session is a logical boundary, not just a network trick. AWS formalizes this in documentation: an AgentCore session is Active while work runs, Idle when provisioned but waiting, and Terminated after a 15-minute idle timeout or an 8-hour maximum lifetime. According to AWS documentation, “an AgentCore session reaches Terminated after a 15-minute idle timeout or an 8-hour maximum lifetime.” That is the lifecycle of a long-running execution environment, not a short-lived HTTP call—and it is the clearest proof yet that session-based compute is now the default for serious agents.
The New Battle Line: Session Isolation Strategies
Once everyone agrees sessions are the core unit, the hard question becomes how to isolate them. The trust boundary forces isolation because agent backends now run code influenced by user input, which must be treated as untrusted. A shared kernel does not provide the tenant isolation that security teams expect for this kind of work. The key difference between platforms is the compute primitive they choose for the isolated environment.
AWS routes each session to its own microVM, creating a VM-isolated sandbox with strong boundaries at the cost of heavier startup and runtime overhead. Google isolates agent code in a sandbox, leaning toward a lighter-weight isolate suited to high-volume automation. Anthropic’s design again emphasizes sandboxes tied tightly to session context. Microsoft’s approach aligns with its Kubernetes bet: attach each agent to a dedicated identity, run it in a session-scoped environment, and let AKS and bare metal options provide performance tuning.
These session isolation strategies are not mere implementation details. They directly shape performance, security, and cost efficiency. A multi-day research agent fits Azure’s longer-lived sessions, a code-heavy agent benefits from microVM-grade isolation, and high-volume workflow automation favors lightweight sandboxes. Many enterprise platforms will mix patterns, but the isolation default—heavy VM, container, or isolate—will define what is economical and what feels safe to run at scale.
From Kubernetes Backbones to Session-Economics
Microsoft’s AKS roadmap shows where this is heading. Its Build announcements frame Kubernetes as the settled home for AI, with the open question now being how to operate AI workloads reliably while balancing cost, performance, and scalability. AKS on Bare Metal cuts out the hypervisor to give AI workloads direct access to NVLink, RDMA, and high-performance networking, a clear bid to reclaim performance lost to virtualization. The result is a model-serving platform that tries to simplify AI deployment without hiding the Kubernetes primitives operators rely on.
At the runtime level, the better mental model is no longer a classic load balancer. It looks more like a virtual actor system, where each session identity is instantiated on demand, kept alive while needed, and torn down when idle, with one live instance per key. The session as a unit of compute ties together state, identity, and isolation. Session-aware execution, once a niche concept, is now the operational unit of enterprise AI infrastructure.
This shift also rewires cloud economics. When billing aligns to active sessions, concurrency, idle time, and agent sizing become the main cost drivers instead of raw request volume. Enterprises that cling to request-centric thinking will overpay for idle capacity or cut corners on isolation. Those that treat sessions as first-class will negotiate better trade-offs: strong isolation where untrusted code runs, lighter sandboxes where throughput matters, and Kubernetes-based backbones that keep the whole system observable and manageable. Session-based compute is not a feature; it is the new substrate. The sooner teams design around it, the less painful the next wave of agents will be.
