LoRA
Fine-tuning by training small low-rank adapter matrices instead of all weights.
Overview
LoRA adapts a model by training a pair of small low-rank matrices alongside each frozen weight instead of updating the weight itself. It solves the cost of full fine-tuning, where every parameter must be stored, updated and shipped per task. It exists because the change needed for a specific task turns out to be low-rank in practice, so a fraction of a percent of the parameters is enough. Multiple adapters can then share one base model.
Technology Graph
How LoRA connects to the rest of the stack.
Applied to
- Axolotl framework Frameworks Axolotl used for LoRA High ✓ source
- Fine-tuning technique Techniques LoRA used for Fine-tuning High ✓ source
- Llama model Models LoRA used for Llama High ✓ source
- Mistral model Models LoRA used for Mistral High ✓ source
- PEFT library LLM Frameworks PEFT used for LoRA High ✓ source
- Qwen model Models LoRA used for Qwen High ✓ source
- Unsloth framework Frameworks Unsloth used for LoRA High ✓ source
Evolution
- QLoRA technique Techniques QLoRA extends LoRA High ✓ source