Local LLMs on Modest GPUs: What the Hype Misses
Running a local LLM GPU setup means hosting large language models directly on your own graphics card for inference, trading cloud dependency for control over latency, data privacy, and long‑term cost while staying within strict VRAM limits that shape which models you can load and how fast they respond.
Many enthusiasts still assume that 8GB of VRAM locks them out of useful local AI, especially for multimodal and coding workloads. That belief is outdated. An older RTX 2070 Super proved capable of handling several practical 8GB GPU AI models, including vision and code assistants, with careful quantization and model choice. At the same time, another user measured real GPU electricity cost on a single RTX 3090 running multiple local LLMs under a controlled workload, using power sampled from nvidia-smi every 10 seconds and priced with a day/night tariff of 0.30 BGN and 0.18 BGN per kWh, converted at 1 BGN ≈ €0.5113. Put together, these two perspectives show the real picture: local AI is accessible and can be cheap, but only if you pick the right models and pay attention to watts, not headline parameter counts.
Quote this if you care about numbers over vibes: “Five of eight local models tested on a single RTX 3090 were cheaper per million tokens than a hosted Flash‑class cloud API on electricity alone”. That is the kind of statement that should shape your next GPU build more than any marketing slide.
Yes, 8GB Cards Can Run Local AI – Including Vision
If you skim modern GPU launches, you still see Blackwell‑ and RDNA 4‑class cards shipping with 8GB of VRAM, a configuration many people write off as inadequate for any AI workload at first glance. Yet real‑world testing on a 2019 RTX 2070 Super showed that 8GB of VRAM is not an excuse to avoid local AI anymore; several models ran well enough to earn a permanent place on that system.
The key is picking models built or quantized for these limits. GLM-4.6V-Flash 9B stands out as the “vision‑capable model 8GB cards desperately needed” because, in a Q4_K_M quantization, it fits comfortably within 8GB while handling screenshots and other images with reliable precision. This matters because multimodal models reserve VRAM for a vision encoder before they even start processing images; on 8GB cards that overhead often decides whether the model stays on GPU or spills into system RAM and crawls. There is a trade‑off: its performance falls off beyond roughly a 16K token context, so it is better suited to shorter sessions than sprawling research chats. But for many privacy‑ or cost‑conscious users, that is a very acceptable restriction in exchange for local image understanding.
Model Choice Affects Your Power Bill More Than You Think
The folk wisdom is that once you own a GPU, local inference is basically free, while the cloud is where the meter runs. That assumption collapses when you start measuring. One enthusiast set up a box named “ardi,” an openSUSE machine with a single RTX 3090 (24GB), and ran local AI models through a generator that sampled GPU power from nvidia-smi every 10 seconds, converted it to kWh, and priced it using real electricity tariffs of 0.30 BGN (day) and 0.18 BGN (night). The result was a direct euros‑per‑million‑tokens figure for each model, with the GPU power curve logged per run.
The lesson: the smallest model is not always the cheapest to run. gemma3:1b on that RTX 3090 was both fast at 136 tokens per second and light at 154W, making it the cheapest overall in energy terms. Yet parameter count alone did not predict cost: five of eight measured models were cheaper than a reference Flash‑class cloud API, while three were more expensive, and they were not the three with the highest parameter counts. GLM-4.5-Air, with 106B parameters, cost €1.040 per million tokens on electricity alone, more than the cloud API before even counting GPU purchase price. DeepSeek-R1-Distill, with 32.8B parameters, ended up the most expensive at €1.526 per million tokens. If you ignore these differences, you can spend more on power than you would have on a hosted model.
Cloud vs Local: When Local LLM GPUs Win on Cost
Cloud AI still has its place, especially for bursty workloads, but the energy measurements from that RTX 3090 tell a different story for sustained use. Five of the eight benchmarked local models delivered a lower energy cost per million tokens than a hosted Flash‑class API reference. The cloud reference used list prices where a Flash‑class model like a Gemini 2.5 Flash or GPT‑4o‑mini tier was around $0.60 per 1M output tokens (about €0.55), and an even cheaper Flash‑Lite tier sat near $0.40 per 1M. Once your usage grows into the millions of tokens, local AI computing can beat that on electricity alone, meaning any further savings on subscription fees or overage charges are all upside.
For cost‑conscious users worried about data privacy, this is where local LLM GPUs shine: you can keep conversations on‑box and predict your long‑run cost with a watt‑meter instead of gambling on future API pricing. The catch is that model choice matters. Energy‑hungry models like GLM-4.5-Air and DeepSeek-R1-Distill can erase any savings by costing more per million tokens than the cloud reference on electricity alone. Careful selection of efficient quantized models, especially in the 1B–9B range, can give you both privacy and measurable savings.
Practical Model Picks and Realistic Expectations for 8GB Cards
For an 8GB gaming‑class card, a practical local stack starts with a small, efficient text model, a vision‑capable assistant, and a lightweight coder. On the coding side, Qwen3.5 4B is described as lightweight and powerful for coding, making it a strong fit for editing and writing smaller code snippets on constrained VRAM. As expected for a 4B model, it cannot be relied on for agentic, multi‑file software engineering tasks, but that is not what you tend to use a 4B model for anyway. For general text and reasoning, Gemma 4 E4B has earned a permanent place on both a modern setup and the RTX 2070 Super test machine, underscoring that you can get solid chat performance from 4B‑class models on 8GB cards without spilling into system memory.
The vision slot is where GLM-4.6V-Flash 9B earns its keep. Quantized to Q4_K_M, it fits within 8GB VRAM, processes short contexts quickly, and can read dozens of screenshots with precise answers. You do have to respect its context window: performance drops beyond about 16K tokens, so it belongs in shorter, focused sessions rather than long‑running research threads. The broader point is simple: once you drop the myth that “8GB is useless for AI,” you can build a capable local toolchain that handles code, images, and chat without any cloud dependency, while keeping an eye on power draw using a meter similar to the RTX 3090 setup that priced every run by real GPU energy.
