llama.cpp
CPU-first inference for quantised models, in portable C++.
Обзор
llama.cpp runs quantised language models efficiently on CPUs, with optional GPU offload, in a codebase with no heavy dependencies. It solves the problem that inference otherwise assumes a datacentre GPU, which excludes laptops, phones and edge devices. It exists because the GGUF quantisation formats it popularised made running a usable model on ordinary hardware practical, and it is the engine behind many local desktop applications.
Граф технологий
Как llama.cpp связана с остальным стеком.
Использует и зависит от
- GGUF infrastructure Infrastructure llama.cpp uses GGUF High ✓ source
- Quantization technique Techniques llama.cpp uses Quantization High ✓ source
Применяется к
- Inference concept Infrastructure llama.cpp used for Inference High ✓ source
- Llama model Models llama.cpp used for Llama High ✓ source
- Mistral model Models llama.cpp used for Mistral High ✓ source
- Qwen model Models llama.cpp used for Qwen High ✓ source
- Speech Recognition application Applications llama.cpp used for Speech Recognition Medium ✓ source