What “three LLMs on one 8GB GPU” really means
Running multiple LLMs on a single 8GB GPU means coordinating GPU memory so that several different language models and agents can share one card without out-of-memory crashes, using a supervising process that schedules model loading, tracks VRAM usage, and rejects requests that would exceed a safe capacity limit while keeping interactive latency acceptable for all workloads.
If you have three AI agents, each wired to a different LLM, a single old GPU usually forces you to pick a favorite and kill the other two. In the typical developer setup, Agent A writes code, Agent B reviews it for security issues, and Agent C drafts documentation at the same time. On an NVIDIA GTX 1080 with only 8 GB of VRAM, launching three separate llama.cpp processes looks fine in tutorials but collapses in reality: the first process reserves a huge KV cache, and the others die with cudaMalloc errors. Your "multi-agent" demo degrades into one agent and two crash logs. The claim that small models fit comfortably is technically true for a single process, and misleading for parallel LLM inference.
Why your second and third LLM crash: KV caches and blind cudaMalloc
The core failure is not that your models are too big; it is that every process pretends it owns the whole GPU. llama.cpp’s llama-completion pre-reserves the KV cache for the full configured context window as soon as the llama_context is created, long before it emits a single token, so decoding never stalls mid-stream. With -c 172032 and -ngl 99, one process happily eats 6,536 MiB out of 8,192 MiB of VRAM before doing any useful work, and most of that footprint is KV cache, not weights.
Once that first context hoards VRAM, the second process walks into a casino. There is no shared queue, no global ledger, only cudaMalloc, which becomes “literally a coin flip the moment the card is past ~80% full”. The result: Agent 2’s Qwen2 0.5B tries to allocate 1,536 MiB and dies; Agent 3’s tiny SmolLM2 360M asks for 5,120 MiB and dies too. Your 8GB VRAM limit feels smaller than advertised because every independent process over-reserves. The problem is not the algorithm, but the lack of GPU memory optimization and shared bookkeeping.
C++ multiplexing: one daemon, three LLMs, no OOM lottery
The practical fix is to stop letting each agent spawn its own GPU-owning process and add a thin C++ multiplexing layer that owns the card on behalf of everyone. lmxd is a single long-lived daemon of about 1,500 lines of C++17 that becomes the sole GPU owner. Agents no longer launch llama-completion binaries; instead, they talk to lmxd over a tiny Unix-socket text protocol (HELP, STATUS, LIST, REGISTER, UNREGISTER). The daemon inspects its internal VRAM ledger, decides whether a new model fits, and only then touches the GPU.
This is classic C++ multiplexing: one process, one llama_backend_init, one refcounted map from GGUF paths to loaded models. If two agents request the same GGUF, the daemon loads it into VRAM once and bumps a reference counter, instead of paying silent driver overhead three times. The model loader and the shared backend turn your ancient GTX 1080 into a small local model deployment platform for parallel LLM inference, without changing any model code. It is not new computer science; it is disciplined bookkeeping written down in C++ and exposed over a Unix socket so a tired engineer can talk to it with nc.
Admission control: the tiny VRAM ledger that keeps everything alive
Multiplexing without admission control is just a nicer way to crash. The real win comes from treating VRAM as a shared, capped budget. In lmxd, the entire policy lives in one number—90% of the card’s VRAM—and one rule: admit a new agent only if currently_used + new_estimate ≤ 90% cap. A small VramLedger tracks allocated_bytes_ and max_vram_bytes_, protects updates with a mutex, and exposes try_reserve(model_table_bytes), which compares-and-bumps in a single critical section so parallel REGISTER calls cannot overcommit.
If try_reserve fails, the daemon returns a structured ERR VRAM_LEDGER_DENY code=CAP line and never touches the GPU. That is the essence of admission control: refuse work before cudaMalloc flips a coin. According to one description, “If the ledger refuses, the daemon returns a structured ERR VRAM_LEDGER_DENY code=CAP line and never touches the GPU”. This simple GPU memory optimization keeps model responsiveness predictable instead of letting latency and OOM errors explode when the card is near full. On the same hardware, with the same three GGUFs—SmolLM2-360M-Instruct-Q4_K_M, Qwen2-0.5B-Instruct-Q4_K_M, and Llama-3.2-1B-Instruct-Q4_K_M—the daemon admits all three without the OOM lottery.
From single-node hacks to serious local deployments
This story is specific—three agents, three LLMs, one aging GTX 1080—but the pattern generalizes to any constrained GPU environment. When you lack high-end GPUs and cannot upgrade “this quarter, or this year, or possibly this lifetime”, a lightweight C++ multiplexer plus admission control is the difference between a toy demo and a reliable local model deployment. The same idea appears at larger scale: an open-source project provides a self-hosted API for post-training and fine-tuning LLMs on standard Kubernetes clusters, abstracting reinforcement learning infrastructure from AI research so machine learning teams can scale post-training workflows on their own clusters. One way it improves efficiency is by running multiple RL jobs to increase overall GPU utilization.
In both the cluster and the lone-desktop cases, specialization matters. Infrastructure-focused tooling lets researchers and application developers work without drowning in system complexity; Google’s engineers describe agentic reinforcement learning as “incredibly easy to get bogged down in system complexity” when concerns are mixed. The multiplexing daemon is the same philosophy in miniature: it decouples GPU ownership and scheduling from agent code. For developers, reliability engineers, and small teams who care about parallel LLM inference but live with the 8GB VRAM limit, the message is clear. Treat VRAM as a shared budget, centralize GPU access in a C++ layer, and enforce admission control. If you do, three different LLMs on one old card stop being marketing and turn into a daily, local reality.
