RLHF
Aligning a model with human preference using reinforcement learning on ranked outputs.
Обзор
RLHF trains a reward model on human comparisons of model outputs, then uses reinforcement learning to steer the language model toward what the reward model scores highly. It solves the problem that the behaviour people want, being helpful and refusing harmful requests, is far easier to rank than to write as a supervised target. It exists because it is what turned a raw next-token predictor into something that follows instructions, and it is the step DPO later simplified.
Граф технологий
Как RLHF связана с остальным стеком.
Использует и зависит от
- Reinforcement Learning concept Algorithms RLHF uses Reinforcement Learning High ✓ source
- DPO technique Techniques DPO based on RLHF High ✓ source
- RLAIF technique Safety & Alignment RLAIF based on RLHF High ✓ source
Применяется к
- Fine-tuning technique Techniques RLHF used for Fine-tuning High ✓ source
- TRL library LLM Frameworks TRL used for RLHF High ✓ source
- Alignment technique Safety & Alignment RLHF enables Alignment High ✓ source
Classification
- Supervised Fine-Tuning technique Techniques Supervised Fine-Tuning part of RLHF High ✓ source
Альтернативы
-
Constitutional AI
technique
Альтернатива для RLHF · source
-
DPO
technique
Альтернатива для RLHF · source
Сравнения
Подробные сопоставления с альтернативами выше.