If you've downloaded a model to run locally – with Ollama, llama.cpp, or anything
else – you've almost certainly run into the letters GGUF.
It's the file extension on most open-weight models you can actually run yourself:
llama-3.2-3b.Q4_K_M.gguf, qwen2.5-7b.Q5_K_M.gguf, and so on.
GGUF is the reason local AI is possible at all. Here's what it actually is, what those
Q4_K_M labels mean, and how to pick the right one for your hardware.
What is GGUF?
GGUF is a file format for storing quantized LLMs – a container that packs a model's weights, tokenizer, and metadata into a single file that llama.cpp and its derivatives (Ollama, LM Studio, etc.) can load directly. It's the successor to GGML, and it's become the de facto standard for running models on consumer hardware.
The clever part is what "quantized" means. A model's weights are normally stored as 16-bit floating-point numbers. GGUF files store them at far lower precision – 8-bit, 4-bit, even 2-bit – which shrinks the file dramatically. That's what lets a 27-billion parameter model fit in 12 GB of VRAM instead of needing 50+ GB.
The trade-off: size vs quality
Quantization is a compression with a price. Lower precision = smaller file + faster inference + lower quality. Higher precision = bigger file + slower + better quality. The skill is finding the sweet spot where quality holds up but the model fits your hardware.
GGUF files encode this in their name. Take qwen2.5-7b.Q4_K_M.gguf:
- Q4 – 4-bit quantization (the "Q" stands for quantization).
- K – "k-quant" (k-quantization), a smarter scheme that uses fewer bits on less important weights.
- M – the size:
S(small),M(medium),L(large). Larger = closer to full quality.
So Q4_K_M means "4-bit k-quant, medium" – the most popular all-rounder,
and the default recommendation for most people.
Common quantization levels
| Level | Approx. size vs full | Quality | When to use it |
|---|---|---|---|
| Q2_K | ~25% | Poor | Only when you must squeeze into tiny RAM |
| Q3_K | ~32% | Meh | Very tight budgets |
| Q4_K_S / Q4_K_M | ~40–44% | Good | Default choice – best size/quality balance |
| Q5_K_M | ~49% | Very good | When you have headroom; notably sharper |
| Q6_K | ~58% | Near original | Plenty of RAM, want quality |
| Q8_0 | ~78% | ~Indistinguishable | Only for the KV cache or when size is no object |
| F16 / F32 | 100% | Original | Rarely needed for inference; huge files |
Which one should you download?
The rule of thumb is simple: start with Q4_K_M. It's the best quality-per-gigabyte, runs on most hardware, and for most tasks the quality difference from Q5 or Q6 is hard to notice. If the model runs comfortably and you have spare RAM, try Q5_K_M and see if you can tell the difference. If it's too slow or doesn't fit, drop to Q4_K_S or Q3_K_M.
To figure out exactly what fits your card, run the numbers through the LLM VRAM Calculator – it accounts for weights, KV cache, and quantization so you know before you download.
GGUF + llama.cpp
GGUF files are designed to be run by llama.cpp – the engine underneath Ollama and LM Studio. You rarely interact with GGUF files directly if you use Ollama (it handles the download and format for you), but the concepts are identical: you're always choosing a model size and quantization. See the llama.cpp guide for the command-line details.
Bottom line
GGUF is the container that makes local AI practical. The letters in the filename tell
you everything: Q4_K_M is the safe default, Q5_K_M is a
worthwhile upgrade if you have headroom, and anything smaller is for squeezing into
tight hardware. Pick Q4_K_M, check it against the
VRAM calculator, and you'll have a local model
running in minutes.