From Requests to Sessions: The New Unit of Compute
Session-based compute for AI agents is an architectural model where the cloud platform treats an entire conversation or workflow as a long-lived, addressable execution context, preserving state, identity, and isolation across many requests instead of handling each call as a stateless, interchangeable transaction. In the past few months, AWS, Microsoft, Google, and Anthropic have all quietly rebuilt their agent runtimes around this idea, treating the agent session—not the individual API call—as the thing the infrastructure schedules and protects. That convergence is not cosmetic; it signals that session-aware execution is becoming the baseline for production-grade AI agents that act more like running processes than like conventional HTTP requests.
The traditional load-balanced model worked because most web APIs assumed each request was independent and any backend could serve it. Stateful AI workloads break both assumptions at once: they need to remember prior turns and they often run code influenced by user input. If you are still thinking of agents as another stateless microservice, you are designing for the wrong failure modes. The new reality is that the agent runtime is a control plane for state, identity, isolation, and lifecycle, and the session is the operational unit it manages.
Why Stateless Request Architectures Fail for AI Agents
Cloud infrastructure design for web applications was built around the idea that state should live outside the worker, in a database or cache, so any replica can serve any request. That design enabled elastic scaling and made it safe to kill and replace servers without losing correctness. Enterprise AI agents refuse to play by those rules. They are long-running, stateful, tool-using processes that often run code shaped by user input.
Consider an agent processing a refund: it reads the order, calls tools, waits for the model, and then handles a follow-up question. If the next turn is routed to a different backend without access to prior context, the workflow breaks. Session affinity and consistent hashing are band-aids that turn the load balancer into a crude session router, but they do nothing about isolation. As platform teams learned from incidents where tenant context was not consistently enforced and roughly 1,000 organizations could see other customers’ project data, correctness now depends on binding user identity, agent context, and session state tightly together. At that point, the session itself becomes the unit of compute the platform must own.
Four Cloud Philosophies on Session Isolation
All four major AI platforms are moving away from treating stateful, untrusted agent work as ordinary load-balanced traffic; they diverge on what they use as the isolated execution primitive. AWS AgentCore is the most opinionated: each session gets a dedicated Firecracker microVM with isolated compute, memory, and filesystem, and requests carrying the same runtimeSessionId are routed back to that microVM via a session header. Those sessions remain Active while work is processed, sit Idle when provisioned but waiting, and are Terminated after a 15-minute idle timeout or an 8-hour maximum lifetime. That is the lifecycle of a long-running environment, not a short-lived HTTP request.
Microsoft rebuilt its Foundry-hosted agents around per-session isolation, creating a VM-isolated sandbox per session that is instantiated on demand and torn down at the end, with each agent assigned a dedicated identity and sessions capped by a 15-minute idle timeout and a 30-day maximum lifetime. Anthropic decomposes its Managed Agents into three parts: a session that logs everything, a harness that runs the loop and routes tool calls, and a sandbox where code runs. The harness behaves as a near-stateless control plane, while the sandbox is a rebuildable resource. Google, meanwhile, isolates agent code in its own sandbox. No single architecture will win outright; code-heavy agents may demand microVM isolation, while high-volume automation will favor lighter sandboxes.
Kubernetes and Fleet Management Meet Session-Based Compute
While agent runtimes are shifting toward sessions, the underlying cloud infrastructure is also changing to host these long-lived, stateful workloads. At its recent developer event, Microsoft unveiled a broad set of enhancements to its managed Kubernetes service aimed at making it a first-class platform for AI training, inference, and large-scale cloud-native applications. Features such as managed system node pools, a lightweight container-focused operating system, and fleet management for multi-cluster estates reinforce a clear view: AI belongs on Kubernetes, and the hard part now is operating AI workloads reliably while balancing cost, performance, and scalability.
Bare-metal Kubernetes support with direct access to NVLink, RDMA, and high-performance networking removes virtualization overhead for GPU-heavy workloads, giving enterprises both operational consistency and near-hardware performance. That matters when the platform is not just routing stateless requests but provisioning and destroying session-aware environments on demand. Traditional load balancers only route traffic to existing backends; the emerging agent control planes provision environments when they first see a session key, keep them alive while they are active or idle, and then tear them down after idle or lifetime limits. This emphasis on managing entire Kubernetes fleets as unified platforms aligns neatly with the idea that the agent session is now the operational unit of enterprise AI infrastructure.
What Developers Should Change in Their AI Agent Architecture
The uncomfortable lesson for developers is that treating agents as glorified HTTP endpoints is no longer defensible. For multi-step reasoning, tool-using, conversational agents, session-based compute is not an optimization; it is essential. The better mental model is closer to a virtual actor runtime, where an addressable identity is instantiated on demand, kept active while needed, and deactivated when idle, with one live instance per key.
In practice, this means designing explicit session lifecycles, choosing isolation strength based on workload, and making state, identity, and tools first-class concerns in your AI agent architecture. A multi-day research agent fits longer-lived sessions, a code-heavy agent deserves microVM-grade isolation, and high-volume automation may rely on lightweight sandboxes. Many enterprise platforms will combine several patterns rather than bet on one. If you are building production AI systems today, ask not where your requests are load balanced, but how your sessions are provisioned, isolated, and terminated. The agent session is becoming the operational unit of enterprise AI infrastructure, and your designs should reflect that.
