llama.cpp
CPU-first inference for quantised models, in portable C++.
Overview
llama.cpp runs quantised language models efficiently on CPUs, with optional GPU offload, in a codebase with no heavy dependencies. It solves the problem that inference otherwise assumes a datacentre GPU, which excludes laptops, phones and edge devices. It exists because the GGUF quantisation formats it popularised made running a usable model on ordinary hardware practical, and it is the engine behind many local desktop applications.
Technology Graph
How llama.cpp connects to the rest of the stack.
Uses and depends on
- GGUF infrastructure Infrastructure llama.cpp uses GGUF High ✓ source
- Quantization technique Techniques llama.cpp uses Quantization High ✓ source
Applied to
- Inference concept Infrastructure llama.cpp used for Inference High ✓ source
- Llama model Models llama.cpp used for Llama High ✓ source
- Mistral model Models llama.cpp used for Mistral High ✓ source
- Qwen model Models llama.cpp used for Qwen High ✓ source
- Speech Recognition application Applications llama.cpp used for Speech Recognition Medium ✓ source