How much VRAM do I need to run an LLM locally?

For a 7B model at 4-bit quantization you want 8 GB; for 13–14B models 12–16 GB; and for 32B models 24 GB or more if you want to fit weights plus the KV cache in memory.

NVIDIA or AMD GPU for local LLMs?

NVIDIA (CUDA) has the smoothest setup with llama.cpp and Ollama. AMD (ROCm/Vulkan) works well too and often gives more VRAM per rupee, but needs slightly more tinkering.

Is CPU-only inference worth it?

For occasional use, yes – modern CPUs can run 7B models at a few tokens per second. For daily use, a GPU with 12 GB+ of VRAM is a much better experience.

What is the cheapest GPU that can run an LLM?

A used 12 GB card (Radeon RX 6800 or RTX 3060 12GB, often under budget in India) runs 7B–13B models at 4-bit quantization. Below that, 8 GB cards handle 7B models with small contexts.

Do I need a different GPU for LLM training vs inference?

Yes – inference just loads the model (VRAM per the size table), while training needs 3–4x more memory for gradients and optimizer states. Nearly all home users only need inference; training happens in the cloud or on workstation cards.

Best GPU for running LLMs locally (2026)

Running an LLM locally is the best privacy/price deal in AI – if you pick the right GPU. Get this wrong and you either buy too much card (wasted money) or too little (your model won't fit). Here's the honest 2026 pick, from budget to enthusiast.

Source note: specs below are from vendor spec sheets; speed numbers are estimates from our GPU AI Performance Calculator unless marked as lab-tested. Lab-tested numbers are from an RX 6800M 12 GB laptop (see lab benchmarks) – your desktop card will be faster, but the fit/offload lessons transfer directly.

🔧 First, check the numbers for YOUR setup

LLM VRAM Calculator – does your target model fit? · GPU AI Performance Calculator – how fast will it run? · Full hardware requirements checklist

Open the calculators →

Quick answer

Best overall: RTX 4090 (24 GB) – still the sweet spot in 2026.
Best value mid-range: RTX 5070 Ti (16 GB) or RX 9070 XT (16 GB) – fast and fit 14B models with room to spare.
Best budget entry: RTX 4060 Ti 16 GB – the 16 GB version specifically. It can run 14B Q4 and even small 30B with offload.
Best Radeon: RX 7900 XTX (24 GB) – most VRAM per rupee on the AMD side.
Best used bargain: RX 6800M (12 GB) or a used 3090 (24 GB) if the price is right.

What actually matters for local LLMs

Three things, in this order:

VRAM is everything. The model plus its KV cache must fit in VRAM or you're offloading to RAM and crawling. This is the hard limit that decides which models you can run at all.
Memory bandwidth sets your speed. Generating text (decode) is memory-bound – the GPU reads the whole model on every token. More bandwidth = more tokens per second.
Compute matters less. Raw TFLOPS only shines during prompt processing (prefill). For chat, a card with lots of bandwidth beats one with lots of compute.

So the buying rule is simple: buy the most VRAM you can afford, then the most memory bandwidth within that VRAM class.

The 2026 lineup, compared

Estimated decode speeds for a 14B model at Q4_K_M (via llama.cpp). Your results vary with context, engine, and cooling.

GPU             VRAM   Bandwidth   Qwen 7B Q4   Qwen 14B Q4   Qwen 27B Q4   ~Price (INR)
────────────────────────────────────────────────────────────────────────────────────────
RTX 4060       8 GB    272 GB/s    ~40 tok/s    offload only   ✗             ₹30,000
RTX 4060 Ti    16 GB   288 GB/s    ~42 tok/s    ~40 tok/s     offload only  ₹48,000
RX 7600 XT     16 GB   288 GB/s    ~42 tok/s    ~40 tok/s     offload only  ₹38,000
RX 7700 XT     12 GB   432 GB/s    ~55 tok/s    offload only  ✗             ₹40,000
RTX 5070 Ti    16 GB   896 GB/s    ~95 tok/s    ~90 tok/s     offload only  ₹80,000
RX 9070 XT     16 GB   644 GB/s    ~72 tok/s    ~68 tok/s     offload only  ₹65,000
RTX 4090       24 GB  1008 GB/s   ~160 tok/s   ~120 tok/s    ~60 tok/s     ₹1,50,000+
RX 7900 XTX    24 GB   960 GB/s   ~150 tok/s   ~115 tok/s    ~58 tok/s     ₹1,20,000

"Offload only" = won't fit fully in VRAM; parts run on system RAM, which is much slower. Numbers are estimates from the GPU AI Performance Calculator.

What lab testing taught us (RX 6800M, 12 GB)

Estimates are useful for shopping; tested numbers keep you honest. On a 12 GB RX 6800M laptop – the weakest card a serious local-AI user should consider – here is what actually happens when the model does not fully fit:

Qwen 27B dense at 3.7bpw (~15.6 GB needed): 18.1 tok/s on ROCm, 21.8 tok/s on Vulkan with partial offload to RAM. Usable for batch work, annoying for chat.
35B MoE with 3B active (Ornith/Tiel-Coder, Q4/Q5 + q8 KV): 25–29 tok/s decode even at 262K context – because only ~3B parameters fire per token. This is why a 16 GB card can feel huge if you pick MoE over dense.
Backend matters on AMD: Vulkan beat ROCm on decode (+20%) in one 27B test; ROCm won prompt eval (+45%) and won decode (+30%) on a 35B MoE. Always test both backends before you blame the card.

Lesson for buyers: if your budget forces partial offload, prefer MoE models with small active parameters and budget for 32 GB system RAM. If you want dense 27B+ fully resident with long context, skip 12–16 GB and go straight to 24 GB. Verify fit first with the LLM VRAM Calculator.

Best overall: RTX 4090 (24 GB)

The 4090 is still the benchmark in 2026. 24 GB of VRAM runs 14B Q4 with a huge context, 27B-class models fully on GPU, and it's the fastest consumer card for prompt processing. The RTX 5090 (32 GB) is faster and fits bigger models, but it costs a fortune – for most people the 4090 (or a used one) is the smarter buy.

Check current price on Amazon →

Best value mid-range: RTX 5070 Ti / RX 9070 XT (16 GB)

If you can't justify a 4090, this is the class I'd buy. 16 GB fits 14B Q4_K_M with 16–32K context comfortably and even 27B Q4 with light offload. The RTX 5070 Ti has almost double the bandwidth of a 4060 Ti – that shows up as roughly 2× the tokens per second. On the AMD side the RX 9070 XT is a solid alternative with strong ROCm support.

RTX 5070 Ti on Amazon → · RX 9070 XT on Amazon →

Best budget: RTX 4060 Ti 16 GB

The 16 GB version is the one to buy for AI. Yes, it's slower than the 8 GB model costs to produce – but the 8 GB version can't run 14B Q4 fully on GPU, while the 16 GB version can, at a still-respectable ~40 tok/s. That's the difference between a toy and a usable local assistant. The RX 7600 XT 16 GB is the Radeon equivalent and usually cheaper.

RTX 4060 Ti 16 GB on Amazon → · RX 7600 XT 16 GB on Amazon →

AMD vs NVIDIA for local AI

The old "AMD is hopeless for AI" advice is outdated. ROCm works well with llama.cpp, Ollama, and LM Studio for RDNA2/RDNA3 cards – see our ROCm installation guide. AMD still wins on VRAM per rupee (the RX 7900 XTX is the cheapest 24 GB card around), while NVIDIA wins on software polish, CUDA libraries, and prefill speed. For a pure local-chat setup, both are fine.

What about the used market?

RX 6800M (12 GB) – a laptop chip that appears on cheap used boards; excellent price/VRAM for 7B–14B work. We've covered it before – it's a great first local-AI card.
Used RTX 3090 (24 GB) – if you find one near ₹60–70k, it's effectively a budget 4090 for AI.
Avoid 8 GB cards for anything above 7B – you'll fight offload constantly.

Used-card failure modes I have seen: 3090s that ran mining workloads with toasted VRAM thermal pads (check VRAM junction temps under load, not just GPU temp), sellers who flashed a different BIOS (verify device ID + VRAM size in GPU-Z), and “16 GB” listings that turn out to be the 8 GB variant. Ask for a 10-minute llama.cpp decode test at the quant you plan to run before you pay. No test, no deal.

Search RX 6800M on Amazon →

Don't forget the rest of the build

VRAM is the star, but a slow CPU or too-little system RAM becomes the bottleneck when you offload. Aim for 16–32 GB system RAM and a decent NVMe for loading model files. Check your whole build with the PSU calculator before you buy that 4090.

Concrete PSU rule: a 4090/7900 XTX class build wants a quality 850W+ unit with native 12VHPWR or 3× 8-pin headroom; 16 GB mid-range cards are comfortable on 650–750W if the CPU is modest. Undersized PSUs do not just crash – voltage droop under sustained decode shows up as random token garbage and driver resets that look like software bugs.

FAQ

Can I run a 70B model on consumer hardware?

Not really on GPU. 70B at Q4 needs ~40–45 GB – that's cloud GPU territory. A 4090 can run it with heavy offload, but it's slow enough to be unusable for chat.

Is 8 GB of VRAM enough?

For 7B Q4 models, yes – barely. For anything bigger, no. That's why every budget pick on this page is 16 GB.

Does NVIDIA CUDA matter that much?

For prefill speed and exotic libraries, yes. For everyday llama.cpp/Ollama chat, the gap has narrowed a lot – Radeon is fine.

What about Apple Silicon?

Unified memory Macs (M-series with 32–64 GB) are genuinely great for local LLMs – that's a separate article. On the desktop side, this page is about NVIDIA/AMD GPUs.

What PSU do I need for a 24 GB card?

850W+ quality unit for 4090/7900 XTX builds, 650–750W for 16 GB mid-range. Run your exact CPU + GPU + drives through our PSU calculator – sustained decode loads punish cheap units.

Is a used RTX 3090 safe to buy?

Yes if you test it: 10-minute decode run, VRAM junction temps, correct device ID/VRAM in GPU-Z, and a return window. Ex-mining cards are fine when cooled properly and priced accordingly – they are not fine at near-new prices with no test.

Bottom line

Buy the most VRAM you can afford, in this order: 24 GB (4090/7900 XTX) → 16 GB (5070 Ti / 9070 XT / 4060 Ti) → 12 GB used (6800M) → anything else is a compromise. Then plug your exact model and context into the calculators below and buy with confidence.

🔧 Work out the fit before you spend

LLM VRAM Calculator + GPU AI Performance Calculator – exact numbers for your model, quant, and context.

Browse all tools →