LLM inference scaling is now the real bottleneck
LLM inference scaling is the set of architectural, infrastructure, and operational strategies used to run large language models at high queries-per-second while meeting latency, reliability, and cost targets for real-world applications in production environments. The hard truth is that the AI story has moved past clever prompt engineering; the real fight is in keeping models online when thousands of requests hit every second. Netflix’s and Databricks’ recent moves show that the winners will be those who treat serving as a first-class problem, not a bolt-on to legacy stacks. Their message is blunt: infrastructure built for traditional batch or low-QPS web workloads is no longer enough to support the traffic patterns of modern, production AI deployment, where every page view and keystroke can trigger inference. Teams that ignore this shift will find their "AI features" stuck in prototype mode, throttled by 429 errors and latency spikes instead of business impact.
Inside Netflix’s model serving platform: Triton, vLLM and hard-earned lessons
Netflix’s in-house model serving platform is a pointed rejection of one-size-fits-all LLM hosting. The company kept its JVM-based serving layer for routing, feature retrieval, candidate generation, post-processing, and logging, then split inference paths by model size and hardware profile. Smaller models run in-process on CPUs, preserving simple deployments, while larger workloads are delegated to MSS, where Triton handles model loading, batching, GPU scheduling, and multi-framework serving. Within the GPU serving infrastructure, vLLM performs inference while Triton controls the environment and scheduling, giving Netflix an extensible engine with stable operational guardrails. That sounds neat on paper, but the production story is harsher: mismatched Triton and vLLM versions can prevent deployments from loading, so compatible releases must be tested and pinned together. Custom models also broke assumptions—Hugging Face compatibility in vLLM was not enough, forcing Netflix to use extension points to support custom architectures and decoding rules. This platform proves that real LLM inference scaling is less about glamorous models and more about lining up the dull details of packaging, compatibility, and GPU sharing.
Reliability gaps: where prototypes die in production
Companies talk about "production-ready" AI, but Netflix’s and Databricks’ experiences show that most prototypes are fragile under real traffic. Netflix discovered that a common serving interface does not remove differences between engines: Triton exposes OpenAI-compatible, KServe HTTP, and gRPC frontends, yet features still behave inconsistently across those integrations. Constrained decoding—forcing valid JSON or other formats by filtering tokens based on generated history—broke when vLLM paused and resumed requests to manage GPU resources, leaving decoder state out of sync. Netflix had to add logic to detect this change and rebuild state before generation continued. On the search side, Databricks describes the familiar failure pattern: 429 (Too Many Requests) errors in production logs, P95 latency spiking as traffic ramps even at moderate utilization, and hacky workarounds like duplicate endpoints or client-side load balancing. These are not edge cases; they are symptoms of architectures built for demos, not for thousands of QPS tied directly to revenue-critical workflows.
Databricks’ high-QPS search: inference scaling by declaration, not wiring
Databricks’ high-QPS scaling for AI Search is an opinionated attempt to hide the traditional pain of capacity planning behind a single parameter. "Standard endpoints can now scale to thousands of QPS with a single, human-readable parameter"—you set target_qps at creation or update it later through the SDK, REST API, or UI, and the platform provisions infrastructure to meet that target. No replica counts, node sizing, or external load balancers. The same endpoint that powered a prototype can handle production traffic, while Unity Catalog governance and Delta Sync remain intact. This model serving platform design directly attacks the friction that used to force bespoke stacks for each search application: manual load testing, discrete index clusters, and constant re-architecting. Search bars, personalization systems, and real-time entity resolution all benefit, especially when every keystroke or page view fires a request. Databricks adds observability—requests per second, latency, and endpoint health—and routes service principal traffic through performance-optimized networks, while personal access token traffic is capped at a few tens of QPS to keep prototypes from quietly masquerading as production.
Serving architecture is becoming the competitive moat
The lesson from Netflix and Databricks is stark: distribution and serving architecture now define competitive advantage as much as model quality. Netflix’s platform lets application teams integrate through a stable surface while models, runtimes, and hosting environments continue to change underneath. That common interface sits above several distinct layers, but abstraction does not remove the underlying work—packaging, compatibility controls, constrained decoding, and deployment isolation still demand careful engineering at each layer. Databricks, meanwhile, is pushing toward an even more opinionated future, where you declare the QPS you need and the system scales automatically. The company is already promising automatic scaling for traffic spikes and support for Storage-Optimized endpoints later this year, all aimed at making search easier to operate at scale. Teams that keep bolting LLMs onto legacy stacks will face mounting reliability debt. Those who treat model serving platforms and GPU serving infrastructure as core products will own the LLM inference economy, not just participate in it.
