RLHF vs Constitutional AI

Aligning a model with human preference using reinforcement learning on ranked outputs.

功能对比

这两项技术尚未记录可对比的属性。

该选哪个？

由上述差异按固定规则生成，不是模型生成的。

Choose RLHF if…

已记录的属性中，没有任何一项显示它更有优势。

Choose Constitutional AI if…

已记录的属性中，没有任何一项显示它更有优势。

关于本页

Only 0 comparable dimension(s); 3 are required before the page is worth indexing.

页面保持可访问，但在积累足够的带来源细节之前不纳入搜索索引（PRD SEO-08）。

RLHF

technique · Techniques

RLHF trains a reward model on human comparisons of model outputs, then uses reinforcement learning to steer the language model toward what the reward model scores highly. It solves the problem that the behaviour people want, being helpful and refusing harmful requests, is far easier to rank than to write as a supervised target. It exists because it is what turned a raw next-token predictor into something that follows instructions, and it is the step DPO later simplified.

Constitutional AI

technique · Safety & Alignment

Constitutional AI has the model critique and revise its own outputs against a written set of principles, then trains on the result. It solves the cost and inconsistency of collecting human judgements for every harmful output: a principle can be stated once and applied to thousands of cases. It exists as an alternative to purely human feedback, with the trade that the principles themselves are authored and therefore carry their authors' judgement.