Local LLM inference is a memory problem, not a hype contest
Local LLM inference is the practice of running large language models on your own hardware instead of a remote cloud, and in the real world it is dominated far more by available memory capacity, model throughput, and electrical cost per token than by headline GPU benchmarks or the latest generation branding.
If you care about running AI at home without burning money on power or subscription fees, the uncomfortable truth is that older and budget GPUs often make more sense than brand‑new flagships. Local AI applications are now easy enough that many people are dropping paid cloud tools and running assistants, code helpers, and agents on their own machines. For this kind of work, a slower card with lots of VRAM can beat a faster, newer one that is starved for memory. The priority is simple: fit the biggest model that meets your quality bar, then chase tokens per second, not marketing claims.
VRAM decides what AI you can run at all
The single most important spec for sustainable local LLM inference is VRAM capacity, because it decides which models you can load without offloading weights to system RAM or disk. Old high‑end GPUs like the RTX 3090 are “crushing” newer cards for local AI because they carry 24GB of VRAM—still a generous amount even compared to recent releases. As one source puts it, “VRAM determines what models you can use”.
This is not an abstract point. A smaller, fast 8B model can answer simple questions or do lightweight text generation, but for heavy reasoning or coding you often need substantially more parameters. When a newer GPU ships with less memory than an older one, you might get more raw compute but you lose access to these larger models. And what good is a more capable chip if it physically cannot hold the model you want to run? For local work, capacity is the gatekeeper; performance is secondary.
Power efficiency is about throughput, not model size
Once a model fits in memory, the real cost story is power per token. The folk wisdom that local LLMs are “free” because you already own the hardware ignores the electricity bill. Careful measurements on both GPUs and Apple Silicon show that cost per token tracks throughput, not parameter count. In other words, watts divided by tokens per second—not model size—is what determines how expensive your local AI habit becomes over time.
On an M3 Ultra Mac Studio with 96GB of unified memory and no discrete GPU, one test compared five models, from a 4B dense model up to a 120B mixture‑of‑experts, all fitted comfortably in RAM. The surprise: the 120B MoE model was about five times cheaper per token than a 27B dense model, despite being much larger. The reason was simple—it decoded more tokens per watt. This same logic applies to GPUs: an older card that runs a medium‑large model efficiently can beat a new flagship that chokes or needs aggressive offloading.
Budget AI hardware: used Tesla V100s, old gaming flags, and integrated chips
Because VRAM and throughput per watt matter more than bleeding‑edge benchmarks, budget AI hardware has become unexpectedly attractive. One enthusiast wanted a single PC for AAA gaming and serious local LLM inference; their RTX 4080 could game fine but lacked memory for bigger models. The solution was delightfully unglamorous: add an NVIDIA Tesla V100 via an SXM2‑to‑PCIe adapter, turning a data‑center accelerator into a home AI co‑processor.
Sourcing a Tesla V100 with 16GB of HBM2 plus the adapter cost about £200, and similar cards can be found for around USD 100 (approx. RM460) on auction sites. That upgrade delivered 32GB of usable VRAM and enough muscle to run a model like Qwen 3.6 at 32 tokens per second. Best of all, for less than USD 300 (approx. RM1,380), you can have your own small to medium‑sized AI models running at home, without any internet connection. This is budget AI hardware in action: not glamorous, but effective.
Dual‑workload PCs and when integrated graphics are enough
PC enthusiasts are getting creative: rather than throw out a gaming rig, they repurpose it as part of a dual‑workload system. In the Tesla V100 build, the user kept the gaming GPU for titles and display output while dedicating the V100—without any display connectors at all—to LLM inference. It took some ingenuity to adapt the SXM module to a desktop board and tame the noisy cooling, but the end result was a gaming PC that can also run 27B‑class models comfortably.
Not everyone needs a discrete card, though. One author measured local LLMs on an M3 Ultra Mac Studio, which has 96GB of unified memory and no separate VRAM pool. All five tested models, up to a 120B mixture‑of‑experts, fit in memory and ran through a local inference server that could be wrapped by their TokenWatt tool. For many people, integrated graphics or Apple Silicon can handle smaller models efficiently, avoiding any GPU spend while still enabling meaningful local assistants and tools.
Stop chasing newest GPUs and start sizing for your models
The pattern is clear: for local LLM inference, VRAM capacity and throughput per watt beat new‑card prestige. Old Nvidia GPUs with large memory pools are still some of the best value options for home labs, precisely because they keep big models resident without spilling to slower system memory. Sourcing a used GPU for AI can be challenging if you lack the budget for workstation parts or top‑tier consumer cards, but mid‑generation flags like the RTX 3090 occupy a sweet spot as more owners upgrade and resell.
In practical terms, the playbook is straightforward: pick the smallest, fastest model that meets your quality needs, then choose hardware that can hold it fully in VRAM or unified memory. A modest, power‑efficient setup will often beat a flashy new GPU on cost per token. If you want sustainable, private AI at home, stop asking “What’s the newest card I can buy?” and start asking “How much memory do my models need, and how many tokens per watt will I get?”
