KV cache offload: the real fix for long-context latency, not more GPUs
GPU-direct KV cache offload is an inference technique where large language model key–value attention caches are written and read directly between GPU memory and shared object storage over RDMA, so long-context prompts can be served without recomputing prefill and without keeping every cache shard in limited high-bandwidth memory, sharply cutting time-to-first-token for repeated or extended-context requests.
The main story here is blunt: if your long-context LLM feels slow, the problem is not a lack of GPU FLOPs, it is that you are recomputing the same context over and over. When inference latency climbs, the standard response has been to buy more accelerators, which “treats a symptom rather than the cause: your GPUs aren’t short of compute, they’re repeating work they’ve already done.” KV cache offload turns that on its head by pushing the cost into shared storage priced per terabyte instead of per accelerator, and recent GPU-direct RDMA support makes that offload fast enough to sit in the hot path of production inference.
Dell now shows that object storage is a first-class target for high-performance KV cache offload, with RDMA landing cache data directly in GPU memory and removing the host bounce buffer from the data path. At 235K-token context this architecture cuts time-to-first-token from 11,223 ms to 837 ms, a 13.4× speedup compared with recomputing prefill. That is not tuning; that is a different architectural choice, and it is one data scientists can adopt through the mainstream vLLM stack instead of a vendor fork.
How GPU-direct RDMA and object storage change the LLM hot path
The bottleneck in long-context inference is the KV cache footprint, not the matmuls. In Dell’s tests, a single 235K-token request produced a 43 GB KV cache, large enough that a typical GPU can hold only a few such requests in HBM and far too valuable to discard and recompute. Without offload, you either evict that cache and pay the prefill bill again, or you strand GPU capacity by reserving memory for historical context. Both options are wasteful.
Object storage fits this problem better than local disk or ephemeral GPU memory. A shareable, multi-instance cache maps cleanly onto an object namespace, and it already co-exists with much enterprise AI data. The missing piece was speed. Earlier, LMCache’s route to object storage still ran over S3-HTTP, with CPU and host memory copies in the data path, so file-based offload had the fast lane. Dell and NVIDIA first solved transport by adding RDMA for S3-compatible storage, bringing zero-copy, kernel-bypass transfers to the S3 API — data moves directly between GPU memory and the storage server, without the HTTP stack in between.
The new step is putting this RDMA path inside the standard inference stack. Both storage engines in the Dell AI Data Platform — PowerScale for file and ObjectScale for object — now carry a GPU-direct RDMA path. With GPU-direct RDMA, offloaded KV cache lands directly in GPU memory via cuObject, so vLLM, LMCache, and NIXL can treat object storage as a “near-GPU” tier instead of a slow archive. Below roughly 8K–16K tokens, offload overhead outweighs the benefit, but at long context sizes this mechanism becomes the obvious hot-path choice.
Upstream vLLM optimization: why this is different from another vendor hack
The most important engineering decision here is not the hardware; it is that the software landed upstream. Dell, working with NVIDIA, contributed an accelerated engine for the NIXL OBJ plugin, with RDMA-based asynchronous PUT/GET, GPU memory (VRAM) segment support, and runtime engine selection. Every enhancement merged upstream rather than being maintained in a private fork, so the optimization becomes part of the common ecosystem instead of a dead-end branch.
Concretely, this means vLLM, LMCache, and NIXL can now offload KV cache to ObjectScale over RDMA, landing directly in GPU memory via cuObject, without any custom client. Dell’s contributions are already present in community releases — LMCache v0.4.5 and NVIDIA NIXL v1.1.0 — with official NIXL builds carrying this support to follow. The core inference stack remains upstream; only the ObjectScale backend configuration is specific to Dell.
For data scientists and platform engineers this matters more than another benchmark chart. You are not being asked to run a bespoke runtime or maintain patches. If you already use vLLM, you can point KV cache offload at ObjectScale and gain GPU-direct, RDMA-accelerated performance from the same open-source stack you trust. In practice, this transforms KV cache offload from a niche optimization feature into an architectural pattern that enterprises, cloud providers, and AI-native builders can adopt without taking on integration debt.
From 11 seconds to under a second: why this changes user-facing LLM design
At long context, the numbers are stark. Dell measured a 13.4× faster time-to-first-token at 235K tokens using GPU-direct KV cache offload versus recomputing prefill: 837 ms compared with 11,223 ms. They also saw 1.3–1.5× better performance than the same offload over S3-HTTP. “837 ms versus 11,223 ms separates an application that feels responsive from one users abandon.” That is the difference between live coding assistance and a tool your developers refuse to use.
This is explicitly a long-context capability. Below about 8K–16K tokens, offload overhead can make TTFT worse — one test showed 91 ms baseline versus 113–129 ms with offload at 4K context. Above that range, the curve flips. If you serve code assistants, document analysis, retrieval-augmented generation (RAG) over large corpora, or agentic workflows carrying long histories, you sit squarely in the sweet spot. Those workloads naturally generate long, frequently repeated prompts that KV cache offload is built for.
The practical impact is that real-time behavior becomes possible at context lengths that used to be offline-only. Instead of trimming history or splitting documents across sessions, you can retain rich context without making users wait double-digit seconds for every follow-up. And crucially, you do it by reusing work, not buying more accelerators: “adding accelerators buys more capacity to redo work; caching removes the work.” For anyone designing user-facing LLM products, that trade-off should reshape how you think about both UX and capacity planning.
What data scientists should do next: design for cache, not brute force
The lesson for practitioners is clear: treat KV cache offload as an architectural pillar, not as a late-stage tuning flag. Long-context LLMs generate KV caches that are too large and too valuable to discard; in one benchmark a single 235K-token request produced a 43 GB cache, and recomputing that every time is little more than setting money on fire. For long-context inference where prefill is being recomputed today, faster storage is more cost-effective than more GPUs, because storage addresses the cause instead of scaling the waste.
In practical terms, if you run RAG, agentic flows, document-heavy analysis, or code assistants, you should design your systems around a shared KV cache tier from the outset. Within Dell’s platform you can pick the engine — PowerScale for file or ObjectScale for object — and in both cases the offload target follows your data architecture rather than dictating it, since each now supports a GPU-direct RDMA path. RDMA for S3-compatible storage is available in ObjectScale 4.3 as a customer preview, with general availability planned in ObjectScale 4.4.
Looking ahead, the wider networking ecosystem is also moving toward fabrics built to keep GPUs busy, with ultra-high-bandwidth, low-latency RDMA and new transports designed for AI workloads. As ultra-large AI clusters grow, the winning designs will be those that combine these fabrics with cache-aware inference stacks. For data scientists, that means the fastest path to better latency is no longer chasing bigger models or more GPUs, but designing pipelines that refuse to recompute the same context twice.
