"This model supports 128K context" sounds great – until you try to run it and wonder where all your memory went. The culprit is the KV cache: the part of an LLM that grows with every token of context, and the reason a big context window costs real gigabytes.
This guide explains what the KV cache is, why it scales the way it does, and how to calculate exactly how much memory a context window needs on your hardware.
What the KV cache is
When an LLM processes a token, it does a lot of computation – and crucially, it needs the results from earlier tokens to make sense of the later ones. Each token's attention to the past is computed from two things per layer: a key (what this token offers) and a value (what it contains).
Rather than recomputing those keys and values for every previous token on every new token – which would make generation quadratically slow – the model stores them. That store is the KV cache. Every token in the context adds a fixed amount to it, across every layer and every attention head.
Why it eats memory linearly with context
Here's the key fact: the KV cache grows with the number of tokens, not the size of the model. More context = more cached keys and values. Double the context and you roughly double the KV cache.
The rough size for a dense model is:
KV cache ≈ layers × kv_heads × head_dim × 2 × bytes_per_value × context
The exact numbers depend on the model architecture. The practical point: for a mid-size model like a 7B or 13B, the KV cache at a large context (say 32K or 128K) can easily match or exceed the size of the weights themselves.
Real numbers
To see this in practice, the LLM VRAM Calculator breaks down weights + KV cache for your exact model, quantization, and context. A couple of worked examples:
- A 7B model at Q4 with 8K context: KV cache is roughly 1 GB – a small slice of the ~4 GB total.
- The same model at 32K context: KV cache jumps to ~4 GB, now matching the weights.
- A 32B model at Q4 with 64K context: the KV cache alone can be 8–12 GB – more than many GPUs have.
That's why "runs at 8K context" and "runs at 128K context" are completely different hardware requirements, even for the same model.
How to reduce it
If the KV cache is eating your memory, you have a few levers:
- Lower the context window (
-c) – the single biggest lever. Use the context you need, not the maximum the model supports. - Quantize the KV cache (
q8_0,q4_0) – storing keys/values at 8-bit or 4-bit roughly halves or quarters the cache, with minimal quality loss for most tasks. - Use MoE or long-context-optimized models – MoE models like the Ornith 35B share KV across tokens, making the cache smaller than a dense model's.
- Offload – llama.cpp can keep part of the KV cache in system RAM, at the cost of speed.
You can see the q8 KV effect in our Qwen 27B Lab test, where q8 KV at 16K fit (just barely) on a 12 GB card with partial offload.
MoE vs dense (why they differ)
A dense model processes every token through all its parameters. An MoE model routes each token through only a few "expert" layers – but the KV cache is a different story. Because attention is shared across tokens regardless of which experts handled them, MoE models can keep a smaller KV cache per token in some configurations. That's one reason a 35B MoE can run at 262K context where a 35B dense model never could. See the MoE vs Dense Lab test for the head-to-head.
Bottom line
The KV cache is the hidden memory cost of LLMs, and it grows with every token of context. When a model says it "supports 128K," that's an upper limit, not a recommendation. Use the context you actually need, quantize the KV cache, and check the VRAM calculator before you commit – and your model will run in a fraction of the memory.