Do I need llama.cpp if I use Ollama?

No – Ollama wraps llama.cpp and handles installation, downloads, and GPU detection for you. Learning llama.cpp is useful for benchmarking and fine-tuning, but not required to run local AI.

How do I speed up llama.cpp?

Offload more layers to your GPU with -ngl 999 if VRAM allows, make sure you built with the right backend (CUDA/ROCm/Vulkan), and use the highest quantization that still fits in memory.

llama.cpp guide: run GGUF models locally

llama.cpp is the engine that Ollama, LM Studio, and most other local AI tools run on under the hood. It's a C++ library that loads GGUF models and runs them on your CPU, GPU, or both, with support for NVIDIA CUDA, AMD ROCm, and Vulkan.

This guide covers the basics: how to install it, how to run a model, how to enable your GPU, and how to read the benchmark output. By the end you'll know exactly what 1713 ms / 373 tok = 4.60 ms/tok = 217.6 tok/s means (and whether it's fast enough).

Installation

The easiest way to get llama.cpp is to build it from source. You need a C++ compiler and cmake:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DLLAMA_METAL=OFF
make -j$(nproc)

For GPU acceleration, add the right flag:

NVIDIA CUDA: cmake .. -DLLAMA_CUDA=ON
AMD ROCm: cmake .. -DLLAMA_HIPBLAS=ON
Vulkan: cmake .. -DLLAMA_VULKAN=ON
Apple Metal: cmake .. -DLLAMA_METAL=ON

If you're on Ubuntu with an AMD card, the ROCm installation guide covers the prerequisites.

Running a model

Download a GGUF file (see GGUF explained for which one to pick), then run:

./build/bin/llama-cli -m path/to/model.q4_k_m.gguf -p "Hello, who are you?" -n 256

This loads the model, runs the prompt, and generates 256 tokens. The output includes timing information that tells you how fast it ran:

1713 ms / 373 tok = 4.60 ms/tok = 217.6 tok/s
eval time = 1713 ms / 373 tok (4.60 ms/tok, 217.6 tok/s)

The first number is prompt eval (how fast the model reads your question). The second is decode (how fast it generates the answer). For chat, decode speed is what matters – it's the stream of text you see. For long documents, prompt eval matters more.

GPU acceleration

By default llama.cpp runs on CPU. To offload layers to your GPU, use the -ngl flag (number of GPU layers):

./build/bin/llama-cli -m model.gguf -p "Hello" -n 256 -ngl 999

-ngl 999 offloads all layers to the GPU. If you have limited VRAM, use -ngl N where N is the number of layers that fit. The LLM VRAM Calculator can help estimate.

For AMD GPUs with ROCm, you may need --load-mode none to avoid splitting weights across GPU and CPU. The Qwen ROCm vs Vulkan benchmark shows how this plays out in practice.

Benchmarking your own hardware

llama.cpp's built-in timing output is all you need. To compare against known results, use the same parameters (model, quantization, context, backend) and note the decode tok/s. Then check the VelsTech benchmark database to see how your hardware stacks up.

Key flags cheat sheet

Flag	What it does
`-m`	Path to GGUF model file
`-p`	Prompt text
`-n`	Number of tokens to generate
`-c`	Context size (e.g. `-c 16384` for 16K)
`-ngl`	Layers to offload to GPU (999 = all)
`-t`	Thread count (default: all cores)
`--repeat-penalty`	Penalize repetition (default 1.1)
`--temp`	Sampling temperature (0.0 = deterministic, 1.0 = creative)

Next steps

llama.cpp is the foundation. Most people never use it directly – they use Ollama, which wraps it in a user-friendly CLI and API. But knowing how llama.cpp works helps you understand what Ollama is doing, and it's essential for benchmarking and fine-tuning. If you're new to local AI, start with the getting started guide – it uses Ollama, which is built on llama.cpp.