Quantization
Storing model weights at lower precision to cut memory and speed up inference.
Обзор
Quantization represents weights, and sometimes activations, at 8-bit or 4-bit instead of 16-bit floating point. It solves the memory and bandwidth cost of inference: a model that does not fit in GPU memory cannot be served at all, and memory bandwidth rather than arithmetic is usually the bottleneck when generating one token at a time. It exists because the quality loss is small for most tasks while the savings are large, which is what makes local inference practical.
Граф технологий
Как Quantization связана с остальным стеком.
Используется в
- FAISS library Vector Databases Quantization used by FAISS High ✓ source
- QLoRA technique Techniques Quantization used by QLoRA High ✓ source
- llama.cpp tool Infrastructure Quantization used by llama.cpp High ✓ source
Применяется к
- Inference concept Infrastructure Quantization used for Inference High ✓ source
- LLM concept Models Quantization used for LLM High ✓ source
- Llama model Models Quantization used for Llama High ✓ source
- Mistral model Models Quantization used for Mistral High ✓ source
- Qwen model Models Quantization used for Qwen High ✓ source
- Unsloth framework Frameworks Unsloth used for Quantization High ✓ source