RLHF trains a reward model on human comparisons of model outputs, then uses reinforcement learning to steer the language model toward what the reward model scores highly. It solves the problem that the behaviour people want, being helpful and refusing harmful requests, is far easier to rank than to write as a supervised target. It exists because it is what turned a raw next-token predictor into something that follows instructions, and it is the step DPO later simplified.
Constitutional AI has the model critique and revise its own outputs against a written set of principles, then trains on the result. It solves the cost and inconsistency of collecting human judgements for every harmful output: a principle can be stated once and applied to thousands of cases. It exists as an alternative to purely human feedback, with the trade that the principles themselves are authored and therefore carry their authors' judgement.
