From Training-Centric Stacks to the Inference Economy
The inference economy is the shift from AI systems focused on training models in controlled environments to AI systems dominated by real-time, large-scale model serving where reliability, latency, and cost per request matter more than raw training throughput. This is not a niche optimization problem; it changes how enterprises design datacenters, software stacks, and financial models for AI. For years, AI infrastructure conversations revolved around training: feeding data to GPUs, checkpointing models, and running long jobs without crashing. That era produced hardware and software tuned for giant, contiguous workloads. Now the money is in inference—answering millions of questions, scoring transactions, and powering agentic systems in production. Most "AI-ready" stacks were sold for the training era; they struggle once real users arrive and demand low-latency, always-on responses. Companies that treat AI inference infrastructure as an afterthought discover performance and reliability problems only when customers complain or churn.
Inference Hits the Datacenter Wall—and Users Feel It
The uncomfortable truth is that the new AI bottleneck isn’t the GPU; it’s the datacenter itself. Datacenter construction declined in 2025 just as AI demand accelerated, and grid connection queues now stretch years. Morgan Stanley projects a 49-gigawatt power shortfall through 2028, which means enterprises cannot expand their way out of inference demand. Every rack unit and kilowatt has to produce more AI work, or margins collapse. When infrastructure falters, ordinary users notice immediately. Payment systems and fraud detection tools rely on AI to make split-second decisions; outages or latency spikes erode trust fast. A model that responds too slowly frustrates users, while one that is tuned too aggressively blocks legitimate purchases. Demo environments, with single requests and clean data, hide this reality. Production brings thousands of concurrent requests from messy devices and networks, and systems that weren’t load-tested against realistic traffic simply buckle. Inference infrastructure now has to be designed explicitly for this volatility and economic pressure.
Netflix: Re-Architecting LLM Serving Platforms for Heterogeneous Hardware
Netflix’s internal LLM serving platform is a clear sign that inference requires a different stack from training. Instead of a monolithic GPU-heavy service, Netflix keeps its existing JVM-based serving layer for routing, feature retrieval, candidate generation, post-processing, and logging. Smaller models run in-process on CPUs, while larger workloads are sent to a separate path where Triton handles model loading, batching, GPU scheduling, and multi-framework serving. Within the GPU flow, Netflix chose vLLM for inference while letting Triton manage the surrounding serving environment. That split allows the company to adopt new inference engines while keeping scheduling and packaging stable. But it introduces new operational realities: mismatched Triton and vLLM versions can prevent deployments from loading, so compatible releases must be tested and pinned together. Custom architectures and decoding behavior also forced Netflix to go beyond generic Hugging Face compatibility, using vLLM extension points to support its own models. This is model deployment optimization under real-world constraints, not slideware.
Uber: Zero Growth Stack as a Template for Inference Cost Reduction
If Netflix shows how to juggle heterogeneous hardware, Uber shows how ruthless you must be about inference cost reduction. Uber’s "Zero Growth Stack" explicitly decouples capacity growth from business demand so services can scale without an ever-growing hardware footprint. Instead of throwing more machines at problems, the company built a scalable infrastructure that optimizes runtime behavior and AI lifecycle management. A central pillar is systematic Go runtime optimization. Uber created GOGCTunner, a library that dynamically adjusts GOGC based on cgroup memory limits and live object utilization. By keeping heap size at roughly 1.25x live objects and enforcing a 70% memory utilization ceiling, the control loop reclaimed 70,000 CPU cores across 30 mission-critical services. At the same time, AI-related costs increased sixfold since 2024, with token-based billing driving individual developer spending up to USD 2,000 (approx. RM9,200) monthly by early 2026. To stop the bleed, Uber moved from unconstrained adoption to strict governance, capping usage at USD 1,500 (approx. RM6,900) per developer and shifting from coarse cost tracking to granular metrics that tie AI-generated code to infrastructure overhead.
What Enterprises Must Change Before the Next 12 Months
The lesson from Netflix and Uber is blunt: your training-era stack will not survive the inference economy without structural change. Inference workloads bring new challenges in model serving, hardware heterogeneity, and cost efficiency that cannot be patched with another GPU purchase. Enterprises need AI inference infrastructure that is tested against realistic traffic patterns, not laboratory benchmarks, so scaling issues surface before users are impacted. According to Liran Zvibel, CEO of Weka, "this is a different problem with a different stack, and most infrastructure currently being sold as ‘AI-ready’ was built for the wrong era." Storage and data layers now have to be judged on how much useful AI work they produce per rack unit, per kilowatt, per dollar. If you are running production AI today or planning to in the next 12 months, your vendor questions must change—from theoretical throughput to proven ability to support high-concurrency inference economics at scale. Companies that move first on realistic load testing, runtime optimization, and model deployment optimization will be the ones whose margins survive the inference wave.
