GGUF
Quantised model format designed for fast loading and CPU inference.
Overview
GGUF packages a quantised model together with its tokeniser and metadata in a single memory-mappable file, so a runtime can load it without a conversion step. It solves the problem that running a model locally otherwise means reconciling a checkpoint, a separate tokeniser and a config that may disagree with both. It exists as the format llama.cpp introduced to make local inference a single file, and most local desktop tools now speak it.
Technology Graph
How GGUF connects to the rest of the stack.
Used by
- llama.cpp tool Infrastructure GGUF used by llama.cpp High ✓ source
Applied to
- Inference concept Infrastructure GGUF used for Inference High ✓ source