GGUF
Quantised model format designed for fast loading and CPU inference.
Überblick
GGUF packages a quantised model together with its tokeniser and metadata in a single memory-mappable file, so a runtime can load it without a conversion step. It solves the problem that running a model locally otherwise means reconciling a checkpoint, a separate tokeniser and a config that may disagree with both. It exists as the format llama.cpp introduced to make local inference a single file, and most local desktop tools now speak it.
Technologiegraph
Wie GGUF mit dem übrigen Stack zusammenhängt.
Genutzt von
- llama.cpp tool Infrastructure GGUF used by llama.cpp High ✓ source
Angewendet auf
- Inference concept Infrastructure GGUF used for Inference High ✓ source