GGUF
Quantised model format designed for fast loading and CPU inference.
Visão geral
GGUF packages a quantised model together with its tokeniser and metadata in a single memory-mappable file, so a runtime can load it without a conversion step. It solves the problem that running a model locally otherwise means reconciling a checkpoint, a separate tokeniser and a config that may disagree with both. It exists as the format llama.cpp introduced to make local inference a single file, and most local desktop tools now speak it.
Grafo de tecnologias
Como GGUF se liga ao resto do stack.
Usado por
- llama.cpp tool Infrastructure GGUF used by llama.cpp High ✓ source
Aplicado a
- Inference concept Infrastructure GGUF used for Inference High ✓ source