How do I run my first model with llama.cpp?

Download a GGUF file, then run llama-cli with -m pointing at the file, -p for your prompt, and -n for how many tokens to generate. The timing lines at the end tell you prompt-eval vs decode speed in tok/s.

CUDA, ROCm or Vulkan – which llama.cpp backend should I choose?

NVIDIA users should build with CUDA; AMD users should try ROCm first and fall back to Vulkan if their card is unsupported. Our ROCm and Vulkan guide benchmarks both backends head-to-head on the same RX 6800M.

Do I need llama.cpp if I use Ollama?

No – Ollama wraps llama.cpp and handles installation, downloads, and GPU detection for you. Learning llama.cpp is useful for benchmarking and fine-tuning, but not required to run local AI.

How do I speed up llama.cpp?

Offload more layers to your GPU with -ngl 999 if VRAM allows, make sure you built with the right backend (CUDA/ROCm/Vulkan), and use the highest quantization that still fits in memory.

How to Run llama.cpp Locally – Install, GPU & GGUF Guide

llama.cpp is the engine that Ollama, LM Studio, and most other local AI tools run on under the hood. It's a C++ library that loads GGUF models and runs them on your CPU, GPU, or both, with support for NVIDIA CUDA, AMD ROCm, and Vulkan.

This guide covers the basics: how to install it, how to run a model, how to enable your GPU, and how to read the benchmark output. By the end you'll know exactly what 1713 ms / 373 tok = 4.60 ms/tok = 217.6 tok/s means (and whether it's fast enough).

How to install llama.cpp (build from source)

The easiest way to get llama.cpp is to build it from source. You need a C++ compiler and cmake:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DLLAMA_METAL=OFF
make -j$(nproc)

For GPU acceleration, add the right flag:

NVIDIA CUDA: cmake .. -DLLAMA_CUDA=ON
AMD ROCm: cmake .. -DLLAMA_HIPBLAS=ON
Vulkan: cmake .. -DLLAMA_VULKAN=ON
Apple Metal: cmake .. -DLLAMA_METAL=ON

If you're on Ubuntu with an AMD card, the ROCm installation guide covers the prerequisites, and our ROCm vs Vulkan guide helps you pick the backend.

How to run your first GGUF model

Download a GGUF file (see GGUF explained for which one to pick), then run:

./build/bin/llama-cli -m path/to/model.q4_k_m.gguf -p "Hello, who are you?" -n 256

This loads the model, runs the prompt, and generates 256 tokens. The output includes timing information that tells you how fast it ran:

1713 ms / 373 tok = 4.60 ms/tok = 217.6 tok/s
eval time = 1713 ms / 373 tok (4.60 ms/tok, 217.6 tok/s)

The first number is prompt eval (how fast the model reads your question). The second is decode (how fast it generates the answer). For chat, decode speed is what matters – it's the stream of text you see. For long documents, prompt eval matters more.

How to enable GPU acceleration (CUDA, ROCm, Vulkan, Metal)

By default llama.cpp runs on CPU. To offload layers to your GPU, use the -ngl flag (number of GPU layers):

./build/bin/llama-cli -m model.gguf -p "Hello" -n 256 -ngl 999

-ngl 999 offloads all layers to the GPU. If you have limited VRAM, use -ngl N where N is the number of layers that fit. The LLM VRAM Calculator can help estimate.

For AMD GPUs with ROCm, you may need --load-mode none to avoid splitting weights across GPU and CPU. The Qwen ROCm vs Vulkan benchmark shows how this plays out in practice.

Benchmarking your own hardware

llama.cpp's built-in timing output is all you need. To compare against known results, use the same parameters (model, quantization, context, backend) and note the decode tok/s. Then check the VelsTech Solutions benchmark database to see how your hardware stacks up.

Key flags cheat sheet

Flag	What it does
`-m`	Path to GGUF model file
`-p`	Prompt text
`-n`	Number of tokens to generate
`-c`	Context size (e.g. `-c 16384` for 16K)
`-ngl`	Layers to offload to GPU (999 = all)
`-t`	Thread count (default: all cores)
`--repeat-penalty`	Penalize repetition (default 1.1)
`--temp`	Sampling temperature (0.0 = deterministic, 1.0 = creative)

Next steps

llama.cpp is the foundation. Most people never use it directly – they use Ollama, which wraps it in a user-friendly CLI and API. But knowing how llama.cpp works helps you understand what Ollama is doing, and it's essential for benchmarking and fine-tuning. If you're new to local AI, start with the getting started guide – it uses Ollama, which is built on llama.cpp.