LLM inference is forcing a new kind of infrastructure
LLM inference optimization is the practice of designing specialized model serving infrastructure, GPU inference scaling strategies, and cost-aware runtime controls so that production LLM deployment can handle high-QPS workloads reliably while cutting AI infrastructure costs compared with naive prototype setups. The uncomfortable truth is that LLMs did not blow up budgets because the models are magical; they blew up budgets because most organizations tried to serve them on data-stack infrastructure that was never designed for interactive, latency-sensitive inference. Netflix, Uber, and Databricks show a clear pattern: if you keep treating LLMs as another analytics workload, you will overpay. If you treat them as a new class of distributed system, you can reclaim massive compute and slash costs by a third or more. That is the real frontier in AI right now.
Netflix: Triton, vLLM and the reality of GPU-first serving
Netflix’s in-house LLM platform is a blunt rejection of the "throw a model behind a REST endpoint" phase of AI. Instead, it layers GPU inference scaling under a stable JVM-based serving interface that still handles routing, feature retrieval, candidate generation, post-processing, and logging. Smaller models run in-process on CPUs, while larger ones are shipped to a separate MSS layer where Triton manages model loading, batching, GPU scheduling, and multi-framework serving. Within that GPU path, Netflix chose vLLM for its extensibility but kept Triton in charge of the serving environment, model lifecycle, and scheduling. The operational message here is opinionated: if you want predictable AI infrastructure costs, you need a clear boundary between the inference engine and the platform around it. Their use of constrained decoding to force outputs like valid JSON underlines another point—serving is not neutral plumbing; it encodes business rules directly into the inference stack.
Uber: Zero Growth Stack and the backlash against runaway AI spend
Uber’s Zero Growth Stack is best read as a response to a painful lesson: uncontrolled AI adoption can wreck a budget long before it delivers value. AI-related costs increased sixfold since 2024, with token-based billing models exhausting budgets and individual developer costs hitting USD 2,000 (approx. RM9,200) monthly by early 2026. Instead of buying more hardware, Uber engineered dynamic system control that decouples capacity growth from business demand, reclaiming 70,000 CPU cores across 30 mission-critical services through runtime optimization such as the GOGCTunner library for Go. By tying garbage collection behavior to cgroup memory limits and live object usage, they turned a static tuning problem into a closed-loop control system. That is not a minor tweak; it is a statement that infrastructure efficiency is now a first-class AI feature. Uber has also shifted from unconstrained usage to strict cost governance with per-developer caps and is auditing AI-generated code to connect throughput improvements to real infrastructure overhead.
Databricks and NBCUniversal: analytics migration as a template for AI economics
NBCUniversal’s migration onto Databricks’ job-specific compute is not about LLMs on the surface, but it reads like a playbook for AI-era infrastructure economics. As data volumes and analytics demands grew, their slot-based reservation model created high compute costs and contention, pushing the company to seek a more scalable, integrated platform. Partnering with Databricks and EXL, they moved to a lakehouse architecture and job-level compute isolation, achieving a 30% reduction in data infrastructure costs. "NBCUniversal achieved a 30% reduction in data infrastructure costs via job-specific compute" is the kind of quotable result AI teams should be aiming for. The deeper lesson is architectural: they unified analytics and ML on one platform, added granular control over compute, and phased cutover only after validation, retiring old pipelines once the new stack proved itself. That same discipline is exactly what LLM inference workloads need—dedicated capacity, clear performance baselines, and migration plans tied to cost metrics, not hype.
The new AI economics: infrastructure as the main lever
Across these stories, a sharp pattern emerges: infrastructure design is now the main lever for AI economics, not model choice alone. Netflix’s layered approach shows how a common serving interface can sit over distinct CPU and GPU paths, giving application teams a stable integration surface even as models and runtimes change. Uber’s Zero Growth Stack proves that dynamic control loops and cost governance can reclaim massive capacity before you buy more hardware. NBCUniversal’s migration shows how granular compute control and unified platforms cut analytics costs and job delays while still scaling around major events. High QPS production LLM deployment cannot survive on prototype-style setups; it demands specialized model serving infrastructure that treats inference as a first-class system with its own architecture, telemetry, and cost model. The companies getting ahead are those treating infrastructure decisions as product decisions: if an AI feature cannot be served cheaply at scale, it is not a real feature yet.
