Two techniques quietly power most fast LLM inference today: Flash Attention and speculative decoding. They're the reason a modern engine can hit speeds that seemed impossible a few years ago. Neither is a new model – they're smarter ways to run existing models.
Here's what they do, how they work, and whether they matter for the local AI you run on your own hardware.
Flash Attention: less memory, faster attention
Attention is the heart of every transformer, but it has a problem: the standard way to
compute it materializes a huge intermediate matrix – seq_len × seq_len –
in fast memory (SRAM), spills it to slow memory (GPU VRAM), then reads it back. That
round trip is pure waste.
Flash Attention restructures the math so the attention scores are computed and consumed in tiles, keeping the working set in fast SRAM and never writing the big intermediate matrix to VRAM. The result:
- Uses far less memory – no more quadratic intermediate matrix, so long contexts fit where they didn't before.
- Is faster – it's memory-bandwidth-bound, and doing fewer slow-memory round trips wins big.
- Gives the same answer – it's mathematically equivalent, just computed more efficiently.
It's the reason modern models can claim 128K–1M context windows at all. Without Flash Attention, that much context would blow past GPU memory in seconds. You don't choose to "use" it – engines enable it automatically when your GPU supports it, and you benefit silently.
Speculative decoding: predict, verify, skip
LLMs generate one token at a time, and each step is a full model pass – serial and slow. Speculative decoding attacks that serial bottleneck with a clever trick:
- A small, fast "draft" model guesses the next few tokens (say 4–8) all at once.
- The big model verifies the guesses in parallel, in a single pass.
- Any tokens the big model agrees with are kept – often 2–4 of them – for the cost of one verification pass.
The win: you get several tokens for roughly the price of one, so effective generation speed goes up – often 1.5–3× – without changing the output distribution. The big model is the source of truth; the draft is just a speed-up.
It works best when the draft model is good at predicting the big model (e.g. a small sibling of the same family). On consumer hardware, llama.cpp and Ollama support draft models, and the speedup is most noticeable for chat where you generate long replies.
What this means for local AI
Both techniques are why modern local AI feels as fast as it does:
- Flash Attention is enabled automatically by llama.cpp / Ollama on supported GPUs (NVIDIA, AMD via ROCm/Vulkan, Apple). It lets you run larger context windows on the same hardware.
- Speculative decoding gives you a free speed boost when you supply a draft model – you keep the big model's quality but generate faster.
- Together with quantization and a managed KV cache, they're the four pillars of practical local inference.
In practice, you don't tune these by hand – the engine does. But knowing they exist explains why the same model can run at very different speeds depending on your backend and flags, which is exactly what we measure in the benchmark database.
How to get the benefit
- Keep your engine up to date – llama.cpp, Ollama, and LM Studio ship Flash Attention and speculative-decoding support in current builds. Old versions are leaving performance on the table.
- Use a modern GPU – Flash Attention needs a GPU with enough SRAM; anything from the last ~5 years is fine.
- Add a draft model for speculative decoding – e.g. a small GGUF of the same family as your main model, and pass it as the draft in your engine's config.
If you're new to local AI and just want to run a model, the getting started guide covers the whole flow; these optimizations happen under the hood.
Bottom line
Flash Attention and speculative decoding aren't magic – they're smarter ways to use the hardware you have. One saves memory and speeds up attention; the other overlaps generation with cheap verification. Both are free performance on modern engines, and both are part of why running serious models locally is practical in 2026.