LLM inference infrastructure is now the real bottleneck
LLM inference infrastructure is the combination of hardware, runtime software, and platform glue that runs large language models in production, turning prompts into responses at scale while meeting strict latency, cost, and reliability targets across heterogeneous CPUs, GPUs, and model sizes. High‑throughput inference is no longer blocked by model quality alone; it is constrained by how well companies design this stack to absorb rising demand without letting hardware and AI bills explode. Netflix and Uber are now treating model serving platforms and hardware efficiency as first‑class strategic assets, and their recent disclosures show a clear pattern: scaling production AI means decoupling business growth from raw compute growth, even if that makes infrastructure more complex.
Netflix: one serving interface, many engines under the hood
Netflix’s in‑house LLM serving platform makes a strong argument: hide complexity from application teams, accept complexity for infrastructure teams. The company extended its JVM-based serving layer so that routing, feature retrieval, candidate generation, post‑processing, and logging stay stable, while inference can shift between CPUs and GPUs with minimal disruption. Smaller models run in‑process on CPUs; larger calls are routed to a separate service where Triton handles model loading, batching, GPU scheduling, and multi‑framework serving. Within that GPU path, vLLM performs inference while Triton retains model-management and scheduling control, a split that favours operational flexibility over purity of architecture. The price is real: Netflix must pin tested Triton and vLLM versions together to avoid backend-loading failures and then layer Red‑Black and versioned deployments on top to isolate incompatible model changes. This is the honest shape of modern GPU serving optimization—abstractions with sharp edges, not magic layers that make problems vanish.
Infrastructure trade‑offs: heterogenous hardware and evolving models
Netflix’s architecture shows the unavoidable trade‑offs when LLM inference infrastructure meets real workloads. Supporting heterogeneous hardware is not optional: the platform must handle CPUs for lightweight in‑process models and GPUs for larger ones while keeping a consistent production workflow as requests move between local and remote hardware. That demands strict control of the serving environment. Triton exposes OpenAI‑compatible, KServe HTTP, and gRPC frontends, but Netflix still saw gaps and behavioural differences across those paths. Meanwhile, rapidly evolving inference engines mean compatibility work never stops. vLLM’s standard Hugging Face support did not cover all custom Netflix models, so engineers used its extension points for bespoke architectures and decoding behaviour. Even constrained decoding—forcing outputs into valid JSON—required extra state management logic when vLLM pauses and resumes requests for GPU resource management. The lesson: in production AI scaling, every abstraction is a contract you must actively police, not a guarantee you can trust once and forget.
Uber: decoupling capacity from demand with the Zero Growth Stack
Uber attacks the scaling problem from the opposite direction: start with cost and hardware efficiency, then fit AI into that frame. Its "Zero Growth Stack" is designed to decouple capacity growth from business demand so the company can reduce its physical hardware footprint while still scaling services. A key pillar is runtime control. Static garbage collection tuning failed for Go services whose memory usage ranged from 100MB to 1GB, so Uber built GOGCTunner, which adjusts GOGC values dynamically based on cgroup limits and live object usage. That automation reclaimed 70,000 CPU cores across 30 mission‑critical services while keeping heap size at about 1.25x live objects and memory utilisation under 70% to avoid OOM events. This is a clear, quotable takeaway: "Automated GC tuning reclaimed 70,000 CPU cores across 30 critical services without breaching a 70% memory utilisation target." It is a direct statement that smarter control loops can replace brute‑force capacity expansion.
AI usage, cost governance, and the next phase of production AI scaling
Uber’s internal generative AI stack shows what happens when model serving platforms succeed: demand explodes, and then the bill arrives. Its platform layers include a model gateway built on Michelangelo AI, an MCP gateway for internal code context, specialised agents such as Minion and Shepherd, and a review layer with uReview and Code Inbox. As a result, 92% of engineers use these agents monthly, 31% of new code is AI‑authored, and Autocover generates over 5,000 unit tests per month. But AI-related costs increased sixfold since 2024, with some developers reaching USD 2,000 (approx. RM9,200) per month by early 2026. Uber responded with strict cost governance, capping usage at USD 1,500 (approx. RM6,900) per developer and, more importantly, shifting metrics from aggregate spend to "net code quality ratio" and "compute efficiency per feature" to decide whether AI‑generated output justifies its infrastructure overhead. This is the next frontier: hardware efficiency and GPU serving optimization tied to business‑level metrics, not raw throughput.
