RLHF vs DPO

Aligning a model with human preference using reinforcement learning on ranked outputs.

功能对比

这两项技术尚未记录可对比的属性。

该选哪个？

由上述差异按固定规则生成，不是模型生成的。

Choose RLHF if…

已记录的属性中，没有任何一项显示它更有优势。

Choose DPO if…

已记录的属性中，没有任何一项显示它更有优势。

关于本页

Only 0 comparable dimension(s); 3 are required before the page is worth indexing.

页面保持可访问，但在积累足够的带来源细节之前不纳入搜索索引（PRD SEO-08）。

RLHF

technique · Techniques

RLHF trains a reward model on human comparisons of model outputs, then uses reinforcement learning to steer the language model toward what the reward model scores highly. It solves the problem that the behaviour people want, being helpful and refusing harmful requests, is far easier to rank than to write as a supervised target. It exists because it is what turned a raw next-token predictor into something that follows instructions, and it is the step DPO later simplified.

DPO

technique · Techniques

Direct Preference Optimization optimises a language model on pairs of preferred and rejected responses in a single supervised-style step. It solves the complexity of RLHF, which needs a separate reward model and a reinforcement learning loop that is notoriously sensitive to hyperparameters. It exists because the RLHF objective can be rewritten so the optimal policy is available in closed form, which removes the need for the reward model and the sampling loop entirely.