LoRA
Fine-tuning by training small low-rank adapter matrices instead of all weights.
Aperçu
LoRA adapts a model by training a pair of small low-rank matrices alongside each frozen weight instead of updating the weight itself. It solves the cost of full fine-tuning, where every parameter must be stored, updated and shipped per task. It exists because the change needed for a specific task turns out to be low-rank in practice, so a fraction of a percent of the parameters is enough. Multiple adapters can then share one base model.
Graphe des technologies
Comment LoRA se rattache au reste de la pile.
Appliqué à
- Axolotl framework Frameworks Axolotl used for LoRA High ✓ source
- Fine-tuning technique Techniques LoRA used for Fine-tuning High ✓ source
- Llama model Models LoRA used for Llama High ✓ source
- Mistral model Models LoRA used for Mistral High ✓ source
- PEFT library LLM Frameworks PEFT used for LoRA High ✓ source
- Qwen model Models LoRA used for Qwen High ✓ source
- Unsloth framework Frameworks Unsloth used for LoRA High ✓ source
Evolution
- QLoRA technique Techniques QLoRA extends LoRA High ✓ source