Quantization
Storing model weights at lower precision to cut memory and speed up inference.
Überblick
Quantization represents weights, and sometimes activations, at 8-bit or 4-bit instead of 16-bit floating point. It solves the memory and bandwidth cost of inference: a model that does not fit in GPU memory cannot be served at all, and memory bandwidth rather than arithmetic is usually the bottleneck when generating one token at a time. It exists because the quality loss is small for most tasks while the savings are large, which is what makes local inference practical.
Technologiegraph
Wie Quantization mit dem übrigen Stack zusammenhängt.
Genutzt von
- FAISS library Vector Databases Quantization used by FAISS High ✓ source
- QLoRA technique Techniques Quantization used by QLoRA High ✓ source
- llama.cpp tool Infrastructure Quantization used by llama.cpp High ✓ source
Angewendet auf
- Inference concept Infrastructure Quantization used for Inference High ✓ source
- LLM concept Models Quantization used for LLM High ✓ source
- Llama model Models Quantization used for Llama High ✓ source
- Mistral model Models Quantization used for Mistral High ✓ source
- Qwen model Models Quantization used for Qwen High ✓ source
- Unsloth framework Frameworks Unsloth used for Quantization High ✓ source