DPO
Aligning a model directly on preference pairs, without training a reward model.
Überblick
Direct Preference Optimization optimises a language model on pairs of preferred and rejected responses in a single supervised-style step. It solves the complexity of RLHF, which needs a separate reward model and a reinforcement learning loop that is notoriously sensitive to hyperparameters. It exists because the RLHF objective can be rewritten so the optimal policy is available in closed form, which removes the need for the reward model and the sampling loop entirely.
Technologiegraph
Wie DPO mit dem übrigen Stack zusammenhängt.
Nutzt und hängt ab von
- RLHF technique Techniques DPO based on RLHF High ✓ source
Angewendet auf
- TRL library LLM Frameworks TRL used for DPO High ✓ source
- Alignment technique Safety & Alignment DPO enables Alignment High ✓ source
Alternativen
Vergleiche
Detaillierte Gegenüberstellungen mit den Alternativen oben.