"What GPU do I need for AI?" is the wrong first question. The right one is "will an LLM run on the machine I already have?" – and the answer is a checklist, not a product pick. VRAM decides whether a model loads; system RAM, CPU, storage, and power decide whether it runs well. This guide walks the full checklist in order, with the numbers from our own VRAM guide and Lab benchmarks on a 12 GB RX 6800M.
If you finish this and decide you need a new card instead, the best GPU for local LLMs picks up where this leaves off. If you want the exact number for your setup, the LLM VRAM Calculator does the math with a fit verdict for your GPU.
Source note: VRAM figures use standard bytes-per-weight (Q4_K_M ~0.6) plus KV/overhead; speed notes are lab-tested on an RX 6800M 12 GB laptop – see lab benchmarks. Prices below are Sept 2026 India street ranges – verify current listings before buying.
The short version
- VRAM first: 8 GB runs 7B, 12–16 GB runs 14B, 24 GB runs 32B – all at 4-bit quantization. Everything else is secondary.
- System RAM second: 16 GB minimum, 32 GB if you ever want CPU offload as a fallback.
- CPU third: 6+ modern cores; it sets prompt speed and carries offloaded layers.
- Storage fourth: model files run 5–40 GB each – a 512 GB NVMe SSD is the practical minimum.
- PSU and cooling last: a loaded GPU draws its full TDP for hours; a tired PSU causes the weirdest "AI crashes."
1. VRAM: the gatekeeper
A model either fits in VRAM or it doesn't – there is no "minimum settings" mode that
makes a 70B model run on an 8 GB card. At the community-standard
Q4_K_M quantization, plus ~1–3 GB for KV cache and overhead:
| Model size | Q4 weights | Total needed | Fits on |
|---|---|---|---|
| 7–8B | ~4–5 GB | ~6 GB | 8 GB card (RTX 4060, RX 7600) |
| 13–14B | ~9–10 GB | ~11 GB | 12–16 GB card |
| 27–32B | ~18–21 GB | ~22 GB | 24 GB card (RTX 4090, RX 7900 XTX) |
| 70B | ~40–45 GB | ~48 GB | Datacenter card or cloud – not home hardware |
Two footnotes that bite beginners: long context eats VRAM (128K context can add 3–6 GB of KV cache – a whole tier), and higher quantization costs VRAM (Q8 roughly doubles the weights column). Our quantization guide shows what you gain for it.
2. System RAM: the safety net
System RAM matters twice: the OS, browser, and engine need their share (8 GB just to breathe), and RAM is where overflow goes when VRAM runs out. 16 GB is the minimum for any local-AI machine; 32 GB if you want partial CPU offload as a fallback – a 30B Q4 can limp along on a 16 GB card with 32 GB of system RAM, at 5–15 tok/s. Slow, but it runs, which beats not running.
3. CPU: speed, not fit
The CPU doesn't decide whether a model loads – but it decides prompt-eval speed (how fast long documents get absorbed) and how painful offload is. Rule of thumb: 6+ modern cores (any Ryzen 5 / Core i5 from the last five years). Below that, even a good GPU feels sluggish on long prompts because the CPU can't feed it. Close browsers and launchers before long prefill runs – a crowded 4-core behaves like a 2-core exactly when the GPU needs feeding most.
4. Storage: bigger and faster than you think
Model files are large and you will collect them: a 7B quant is ~5 GB, a 32B quant ~20 GB, and you will keep three "just to compare." 512 GB NVMe minimum, 1 TB comfortable. Speed matters at load time (NVMe vs SATA is seconds vs a minute per model load) and for offload swap – never put models on a spinning drive you also boot from.
Two details buyers skip: keep 15–20% SSD free or write speeds crater exactly when offload swap needs them, and expect models to churn – weekly new quants mean hundreds of GB written monthly, so a DRAM-equipped NVMe with real endurance rating outlasts bargain QLC for a model library that never stops growing.
5. PSU, cooling, OS: the unglamorous trio
- PSU: a GPU at full inference load draws near its TDP for hours. A 650W quality unit for mid-range cards, 850W+ for 24 GB flagships. Random crashes under load are a dying PSU until proven otherwise.
- Cooling: sustained 100% GPU utilization is hotter than gaming bursts. Undervolting a Radeon costs ~5% speed for ~20% less heat – worth it on laptops especially.
- OS and drivers: Linux + ROCm for AMD, any OS + CUDA for NVIDIA. Our ROCm vs Vulkan guide and ROCm install guide cover the AMD path step by step.
Three machines, verdicts
- Thin-and-light laptop (integrated graphics, 16 GB RAM): 7B Q4 via CPU at a few tok/s, or small models only. Usable for chat, painful for agents.
- 2021 gaming laptop (RTX 3060 / RX 6800M, 12 GB VRAM, 32 GB RAM): the sweet spot – 14B fully, 27–32B with tuned quants (our Lab runs exactly this class daily).
- Desktop with a 24 GB card: 32B fully with long context. The point where local stops feeling like a compromise for most work.
The MoE exception (why 35B sometimes beats 32B on small cards)
Dense-model math says 35B needs more VRAM than 32B – true for weights (~20 GB at Q4 either way), misleading for experience. A 35B MoE with ~3B active parameters (Ornith, Tiel-Coder) decodes like a 7B: our lab held 25–29 tok/s at 262K context on a 12 GB card with partial offload, while a 27B dense at the same quant managed 18–22 tok/s at 16K. If your card forces offload, a 30–35B MoE with small actives is the best quality-per-GB you can buy – verify weights fit plus KV at your target context in the calculator rather than assuming “bigger number = needs bigger card”.
What to upgrade first (bottleneck order)
- Model doesn't load at all → VRAM. No other upgrade substitutes. Drop a quant tier, shorten context, or buy VRAM – in that order of cheapness.
- Loads but crawls on long docs → system RAM + context discipline. 32 GB RAM plus halved context beats a new GPU for KV-bound slowness.
- Loads, short context fine, long prompts stall → CPU. Prefill is CPU-fed; 6+ modern cores is the floor, and background apps stealing cores show up as “GPU is slow”.
- Random crashes under sustained load → PSU + cooling. Check wattage headroom in the PSU calculator, watch VRAM junction temps (not just GPU temp), and undervolt before replacing.
- Everything works but model loads take forever → NVMe. Last priority, first quality-of-life win – seconds versus a minute per swap when you keep three models around.
India buying notes (Sept 2026 ranges)
Street reality: 8 GB cards (4060, 7600) ~₹27–35k; 16 GB mid-range (4060 Ti 16 GB, 7600 XT) ~₹38–48k; 24 GB used 3090s near ₹60–70k when tested (VRAM thermals + device-ID check – see used-market checks); 24 GB new flagships far above. Pair any 16 GB+ card with 32 GB system RAM (₹8–12k for a DDR4/DDR5 kit) before chasing a bigger GPU – RAM is the cheapest offload insurance. Laptops: prefer 32 GB unified/RAM configs at purchase; soldered 16 GB with no slot is a local-AI dead end you cannot upgrade out of.
Bottom line
Check in order: VRAM → RAM → CPU → storage → PSU. Most "can I run local AI?" questions are answered at step one – 8 GB means 7B, 16 GB means 14B, 24 GB means 32B. Run your exact model through the VRAM calculator before spending money, and only then ask which card to buy. Print this order and tape it to the monitor before opening any store page.
FAQ
How much VRAM do I need to run an LLM?
Q4: ~6 GB total for 7B (8 GB card), ~11 GB for 14B (12–16 GB card), ~22 GB for 32B (24 GB card). 70B needs ~48 GB – cloud, not home.
How much system RAM do I need for local AI?
16 GB minimum; 32 GB if you want CPU offload fallback – a 30B Q4 limps along on a 16 GB card with 32 GB RAM at 5–15 tok/s.
Does CPU matter for running LLMs locally?
Not for fit – VRAM decides. It sets prompt-eval speed and offload pain: 6+ modern cores is the rule; below that long prompts feel sluggish.
Can I run a 70B model at home?
Not practically – ~48 GB at Q4 exceeds every consumer card. Multi-GPU or cloud; 32B on 24 GB is the realistic home ceiling.