GGUF
Quantised model format designed for fast loading and CPU inference.
Обзор
GGUF packages a quantised model together with its tokeniser and metadata in a single memory-mappable file, so a runtime can load it without a conversion step. It solves the problem that running a model locally otherwise means reconciling a checkpoint, a separate tokeniser and a config that may disagree with both. It exists as the format llama.cpp introduced to make local inference a single file, and most local desktop tools now speak it.
Граф технологий
Как GGUF связана с остальным стеком.
Используется в
- llama.cpp tool Infrastructure GGUF used by llama.cpp High ✓ source
Применяется к
- Inference concept Infrastructure GGUF used for Inference High ✓ source