The new pattern: database AI inference, not database-to-API chat
Database AI inference is an architecture where lightweight AI models execute directly inside the database engine or close to it, so that queries involving semantic filtering, ranking, or prediction run at database speed without sending per-row calls to external large language model APIs or separate ML services.
The key shift in AI infrastructure is simple: databases are tired of being chatty clients, and are starting to become AI runtimes in their own right. Instead of shipping rows to remote large language models over HTTP, engines are deploying local model deployment strategies that keep computation next to the data. Google’s AlloyDB AI functions now let developers call LLM-style capabilities like ai.generate, ai.if, ai.rank, ai.forecast, ai.summarize, ai.agg_summarize, and ai.analyze_sentiment directly from SQL queries. That alone is a big usability step—but the opinionated takeaway is stronger: if you are still doing per-row LLM calls from your application tier, your architecture is already behind.
According to one vendor’s internal testing, smart batching lifts throughput by 2,400x, and optimized proxy models reach 23,000x throughput with a 6,000x reduction in cost compared to a row-at-a-time baseline. Even if you discount the marketing math, the direction of travel is clear: stop paying network and token penalties for work that should live inside the data layer.
LLM proxy models: the database learns, then runs the model locally
AlloyDB’s proxy models are the clearest sign that the relationship between databases and LLMs is being inverted. The old pattern was a database that emits rows to an external model for every semantic decision. The new pattern turns the external LLM into a teacher that trains a smaller student model living directly in the database. First, a PREPARE statement sends a sample of your data to a frontier LLM and uses the outputs to train a lightweight local model inside the database. Then, EXECUTE runs the query using this LLM proxy model instead of calling the external service.
That change matters far more than any single benchmark. Now, semantic filters like ai.if can be pushed down into the storage engine and evaluated at database speed on large tables. AlloyDB reports throughput of 100,000 rows per second when using these proxy models. The system also keeps a safety net: if the proxy’s confidence is too low or no proxy is trained, execution falls back to the frontier model. The architectural bet is clear: train once on a representative slice, then amortize that knowledge over millions of rows locally. Any database still shipping every row across the network for classification will lose on both cost and latency.
Smart batching: the minimum bar for AI-powered queries
Even before you adopt proxy models, smart batching has become the price of entry for serious AI-powered database workloads. Calling an LLM once per row is a structural mistake: in a table with 100,000 products, that means 100,000 round trips, 100,000 copies of the same system prompt, and 100,000 times you pay per-token inference costs. Smart batching turns this pattern on its head by grouping multiple rows into a single external model call and sending the shared prompt only once.
For functions like ai.if and ai.rank, this approach drives throughput up to 10,000 rows per second in internal tests, a 2,400x improvement over row-at-a-time processing. The blunt conclusion: if your stack cannot batch AI calls from the database, you are wasting money and blocking your own analysts. Because AlloyDB’s AI functions behave like standard SQL operators, they allow queries to filter rows by meaning rather than brittle keyword matches. That is exactly the kind of semantic power users expect from “AI in the database” — but without batching and proxy models, those queries will be too slow or too expensive to use in production.
GPU-accelerated queries: Presto shows what “database speed” will mean
Local model deployment is only half the story; the other half is raw query speed. GPU-accelerated engines such as Presto running on modern NVIDIA hardware display where this is heading. On NVIDIA GPUs, Presto delivers peak performance for analytical query workloads and provides low latency for both users and agents. A single DGX B200 node with eight GPUs runs eight Presto GPU workers connected with NVLink, and this GPU-accelerated Presto configuration shows clear performance advantages over a multi-node CPU cluster based on Intel Xeon 6642Y servers.
At a 1 TB scale factor, GPU-accelerated Presto delivers between 2.5x and 8x lower query latency than the CPU-only cluster, depending on how many GPUs are active. Those gains are not academic. Whether teams are running interactive dashboards or nightly batch jobs, this GPU-accelerated Presto setup provides low latency and high throughput, which is exactly what AI-driven query plans will demand. The roadmap is explicit too: the maintainers plan to improve the throughput of sending results from GPU workers to the coordinator to keep GPU utilization high without query rewrites, and are offering access to a technical preview of GPU-accelerated Presto on top of a data lake platform. In other words, the compute side of “database AI inference” is already being tuned for the next wave of in-database models.
What this architectural shift means for enterprise teams
The direction of travel is unmistakable: AI logic is moving down-stack, towards the data and the query engine. Any database that calls external models for per-row decisions faces the same cost and latency wall. Proxy models and smart batching are the emerging escape hatches. The practical implication is that AI architects should start from the database and query engine when designing semantic search, classification, and ranking features, not from a standalone LLM API.
In the near term, expect hybrid setups: frontier models for training and edge cases, lightweight in-database models for the hot path, and GPU-accelerated queries to keep analytics and inference in the same tight loop. GPU-accelerated Presto already brings low latency to analytical workloads, keeping both humans and agents unblocked and able to iterate rapidly. The question for most enterprises is no longer whether they will do AI inside the database, but how aggressively they are willing to redesign schemas, workflows, and governance to take advantage of it. Those who hesitate will find that their AI features either cost too much to run or are too slow to matter.
