What Local AI Inference Is—and Why Latency Matters
Local AI inference is the process of running neural networks directly on personal devices—phones, laptops, workstations, and edge servers—so models generate answers without sending data to remote cloud APIs, which reduces latency, strengthens privacy, and allows applications to keep working even when network connectivity is weak or unavailable. In practice, this shifts computation closer to users, turning GPUs and mobile chips into real-time assistants rather than thin clients for cloud models. Latency becomes the key user-facing metric: shaving hundreds of milliseconds per response can change a chatbot from stuttering to conversational, or make autocomplete feel instant instead of distracting. Two new moves from Google and NVIDIA target exactly this bottleneck. Both focus on smarter decoding strategies and optimized runtimes so local hardware can generate many more tokens per second while consuming less memory and avoiding wasteful data transfers.
Inside Google’s LiteRT-LM and Gemma 4 Multi-Token Prediction
Google’s LiteRT-LM framework brings native support for Gemma 4 Multi-Token Prediction drafters, using speculative decoding to turn local AI inference into a much faster loop. Instead of predicting one token at a time, Gemma 4’s multi-token prediction (MTP) drafters propose several future tokens in parallel, which the main model then verifies in a single pass. According to Google, this makes MTP decoding "1.6x faster for Gemma 4 E2B and 2.2x faster for Gemma 4 E4B" while staying on-device. LiteRT-LM builds on LiteRT (formerly TensorFlow Lite) with an orchestration layer tuned for large language models, including shared KV cache handling, memory locality, and reduced CPU–GPU data transfers. Advanced quantization, XNNPACK and MLDrift kernels, and aggressive session management keep prefill and decode performance competitive with or ahead of llama.cpp, MLX, Cactus, and ONNX for on-device AI performance.
From Kotlin to Swift: Framework Support Reaches Phones and the Web
One reason LiteRT-LM matters is its growing support across languages and platforms, which lowers the barrier to shipping local AI inference inside everyday apps. The framework already supported Kotlin and C++, and is now expanding to Swift and JavaScript APIs, opening the door to native iOS apps and browser experiences that tap Gemma 4 locally. Under the hood, LiteRT-LM focuses on memory efficiency: it keeps per-layer embeddings out of memory and loads image and audio encoders only when needed, so even the roughly 2.58GB Gemma 4 E2B model can run in about 607MB on Apple mobile CPUs. Built-in session management lets apps save and restore KV cache state, enabling long, multi-turn interactions without recomputing history. Combined with support for constrained decoding, function-calling, and Gemma 4 "Thinking Mode", these features push on-device AI performance from simple text completion toward full agentic workflows on phones and browsers.
NVIDIA Brings DiffusionGemma to RTX and DGX at High Throughput
On the GPU side, NVIDIA is rolling out full DiffusionGemma support across its RTX and DGX platforms, targeting fast local AI inference for text and image workloads. DiffusionGemma is built on Google’s Gemma 4 mixture-of-experts architecture and combines it with a diffusion head that can denoise up to 256 tokens per step instead of predicting them sequentially. NVIDIA states that this enables "roughly 4 times faster performance than an equivalent autoregressive model" on its hardware. With tensor cores and the CUDA stack, NVIDIA reports throughput figures such as 1000 tokens per second on H100 GPUs in DGX Stations and 150 tokens per second on DGX Spark systems, with up to 800 tokens per second for DGX Station in local inference scenarios. Day-one DiffusionGemma support in Hugging Face Transformers, vLLM, and Unsloth means developers can move from experiments to real deployments more easily.
What Faster On-Device AI Means for Privacy and Real Apps
Multi-token prediction is the common thread across Gemma 4’s MTP drafters and DiffusionGemma. By drafting many tokens at once—whether via speculative decoding or diffusion-style denoising—models cut the number of inference steps they need to produce a response. That translates into shorter wait times, smoother UI, and better on-device AI performance without relying on cloud capacity. For end users, more of the interaction happens on local hardware, which reduces the need to send sensitive prompts or context over the network and lowers exposure to server-side logging. For developers, LiteRT-LM’s mobile and web focus plus NVIDIA’s RTX and DGX coverage mean they can design agentic systems—tool-calling, long conversations, structured outputs—that run close to users. As local AI inference 2x–4x speeds up, product decisions will shift from "can this run locally at all?" to "when does it make sense not to run it locally?"
