llama.cpp
CPU-first inference for quantised models, in portable C++.
Aperçu
llama.cpp runs quantised language models efficiently on CPUs, with optional GPU offload, in a codebase with no heavy dependencies. It solves the problem that inference otherwise assumes a datacentre GPU, which excludes laptops, phones and edge devices. It exists because the GGUF quantisation formats it popularised made running a usable model on ordinary hardware practical, and it is the engine behind many local desktop applications.
Graphe des technologies
Comment llama.cpp se rattache au reste de la pile.
Utilise et dépend de
- GGUF infrastructure Infrastructure llama.cpp uses GGUF High ✓ source
- Quantization technique Techniques llama.cpp uses Quantization High ✓ source
Appliqué à
- Inference concept Infrastructure llama.cpp used for Inference High ✓ source
- Llama model Models llama.cpp used for Llama High ✓ source
- Mistral model Models llama.cpp used for Mistral High ✓ source
- Qwen model Models llama.cpp used for Qwen High ✓ source
- Speech Recognition application Applications llama.cpp used for Speech Recognition Medium ✓ source