llama.cpp
CPU-first inference for quantised models, in portable C++.
Überblick
llama.cpp runs quantised language models efficiently on CPUs, with optional GPU offload, in a codebase with no heavy dependencies. It solves the problem that inference otherwise assumes a datacentre GPU, which excludes laptops, phones and edge devices. It exists because the GGUF quantisation formats it popularised made running a usable model on ordinary hardware practical, and it is the engine behind many local desktop applications.
Technologiegraph
Wie llama.cpp mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- GGUF infrastructure Infrastructure llama.cpp uses GGUF High ✓ source
- Quantization technique Techniques llama.cpp uses Quantization High ✓ source
Angewendet auf
- Inference concept Infrastructure llama.cpp used for Inference High ✓ source
- Llama model Models llama.cpp used for Llama High ✓ source
- Mistral model Models llama.cpp used for Mistral High ✓ source
- Qwen model Models llama.cpp used for Qwen High ✓ source
- Speech Recognition application Applications llama.cpp used for Speech Recognition Medium ✓ source