GGUF
Quantised model format designed for fast loading and CPU inference.
Aperçu
GGUF packages a quantised model together with its tokeniser and metadata in a single memory-mappable file, so a runtime can load it without a conversion step. It solves the problem that running a model locally otherwise means reconciling a checkpoint, a separate tokeniser and a config that may disagree with both. It exists as the format llama.cpp introduced to make local inference a single file, and most local desktop tools now speak it.
Graphe des technologies
Comment GGUF se rattache au reste de la pile.
Utilisé par
- llama.cpp tool Infrastructure GGUF used by llama.cpp High ✓ source
Appliqué à
- Inference concept Infrastructure GGUF used for Inference High ✓ source