From LLM API Calls to Database-Embedded AI
Moving AI inference inside the database means training lightweight local inference models from large language model outputs so that semantic filtering, ranking, and search run at database speed instead of relying on slow, costly external LLM API calls. This shift turns the database into an AI runtime, cuts per-request overhead, and makes semantic applications depend on local computation rather than remote model endpoints. This is not a minor performance tweak; it is a power shift. When every semantic query requires an external model call, your AI stack is held hostage by network hop latency, per-token billing, and rate limits. When inference is embedded, semantic search scaling, vector database optimization, and LLM cost reduction become database problems rather than cloud API problems. The emerging pattern across vendors is clear: use frontier LLMs as teachers, then run distilled decisions locally for day‑to‑day work.
AlloyDB’s Proxy Models: LLM as Teacher, Database as Student
Google’s AlloyDB AI functions make the clearest statement yet: the database should learn from the LLM once, then stop calling it for every row. Developers can now embed AI into standard SQL with operators like ai.generate, ai.if, ai.rank, ai.forecast, ai.summarize, ai.agg_summarize, and ai.analyze_sentiment, so queries can filter and rank by meaning instead of keywords. The problem AlloyDB targets is painfully familiar. A table with 100,000 products means 100,000 round trips to a remote model, each repeating the same system prompt and racking up per-token costs. Smart batching attacks this first: it groups many rows into a single model call, delivering up to a 2,400x throughput boost and pushing about 10,000 rows per second in tests. That is helpful, but still leaves you dependent on an external service. The proxy model architecture is the real break with the past. A PREPARE statement sends a sample of data to a frontier LLM and uses its responses to train a lightweight local inference model inside AlloyDB. Later EXECUTE calls use ai.if with USING proxy to run semantic filters at database speed, falling back to the remote model only when the proxy lacks confidence or does not exist. Google reports throughput of 100,000 rows per second using this pattern and a 23,000x throughput improvement with a 6,000x reduction in cost for the previewed optimized proxy model. According to Google’s internal testing, “smart batching delivers a 2,400x improvement in throughput over row-at-a-time processing, while optimized proxy models push that to 23,000x with a 6,000x reduction in cost.” That headline deserves skepticism and benchmarking, but the architecture is sound: turn the LLM into a teacher, not a permanent runtime dependency.
HubSpot: Scaling Semantic Search by Owning the Vector Layer
While AlloyDB shows how to embed AI into SQL, HubSpot’s story shows why enterprises are refusing to outsource vector infrastructure. Its Vector as a Service (VaaS) platform has grown from a proof of concept into an internal system managing more than 20 billion vectors across over 38 teams, backing agents, retrieval‑augmented generation (RAG), and contact deduplication. As agent usage climbed, retrieval quality and latency stopped being nice‑to‑have metrics and became business‑critical. HubSpot built VaaS on top of Qdrant, chosen because it can run on‑premises, supports named vectors, hybrid search, multi‑stage querying, weighted reranking, and offers cost controls like quantisation and on‑disk storage. That is vector database optimization in the real world: not blog‑post benchmarks, but choosing features that keep recall high while taming storage and compute. The scale is serious: more than 200 indexes, 140‑plus clusters, five regions, two environments, and write traffic that peaks at 100,000 requests per second. Manual operations broke long before this. Their early Helm‑based setup could not call APIs, auto‑scale from external metrics, or handle complex lifecycle work; as they put it, “Manual operations don't survive growth.” They moved to an internal Kubernetes Operator framework and a Translator reconciliation loop that continuously aligns desired and actual cluster state, automating shard movement, replication recovery, and cluster creation and teardown. The payoff is operational, not cosmetic. Cluster spin‑up time dropped from hours to minutes, standby clusters were removed, and the same reconciliation model now handles horizontal scaling and shard rebalancing as collections grew to billions of points. When one imbalanced shard can force an entire cluster to scale early, owning the vector layer is the only responsible way to do semantic search scaling at this magnitude.
Smart Batching and Local Inference: The Real-Time Semantic Stack
Put AlloyDB’s proxy models next to HubSpot’s VaaS and a new pattern appears: real‑time semantic systems will be built from smart batching, database‑embedded AI, and in‑house vector infrastructure. Smart batching turns naive row‑at‑a‑time LLM calls into high‑throughput workflows by squeezing many items into one request, sending the system prompt once and the data in bulk. For ai.if and ai.rank this reaches roughly 10,000 rows per second in internal tests, solving the first layer of latency and LLM cost reduction. Proxy models then go further by training local inference models so that most semantic decisions run inside the database, at the same speed as traditional filters. On the retrieval side, vector databases must be treated as operational systems, not toys. Qdrant’s own guidance focuses on tuning for hundreds of millions of vectors while staying within production latency and accuracy budgets. Other vendors echo the same theme: recall is king, storage and compute should be separated, reliability is a feature, and indexes should be versioned and filter‑aware. HubSpot’s emphasis on balanced shard placement and avoiding memory skew is another way of saying the same thing: semantic search is only useful if it is fast enough and predictable enough for agents and RAG pipelines to rely on. Combined with AlloyDB’s vector search powered by Google’s ScaNN index, supporting up to 10 billion vectors, structured queries, semantic search, and LLM‑powered analysis now coexist inside the same SQL layer. That convergence is not optional; it is the only way to get real‑time semantic behaviour without drowning in external model calls.
What Enterprises Should Do Next
The lesson from both AlloyDB and HubSpot is blunt: if you keep calling frontier LLMs for every semantic decision, your AI stack will stall under its own economics. Enterprises should start by deciding which decisions need frontier‑grade reasoning and which can be handled by local inference models trained from samples. AlloyDB’s proxy pattern offers a blueprint: treat per‑row filters as learned judgments, store them near the data, and reserve direct LLM calls for high‑value or low‑confidence cases. Since its optimized proxy model for ai.if is in preview and smart batching requires an explicit database flag, teams must benchmark with their own data distributions and workloads before trusting marketing figures. At the same time, semantic search scaling demands operational discipline. HubSpot’s experience shows that building an internal Vector as a Service, coupled with Kubernetes Operators and continuous reconciliation, is not over‑engineering but survival at tens of billions of vectors and 100,000 write requests per second. Owning your vector database optimization, access control, embeddings generation, and cost tracking keeps customer data under your control and makes AI behaviour explainable to regulators and users. The opinionated path forward looks like this: use frontier LLMs as teachers, not permanent workers; embed AI into databases where possible; treat vector infrastructure as a core platform, not a sidecar service. Teams that keep AI logic inside the database will gain lower latency, fewer external dependencies, and more predictable costs. Those that continue to scatter semantic logic across brittle LLM APIs will spend the next few years firefighting outages and explaining their cloud bill.
