TRL
Training library for fine-tuning and aligning models on preference data.
Overview
TRL provides the training loops for supervised fine-tuning, reward modelling and the preference optimisation methods that follow. It solves the problem that aligning a model is a pipeline of stages with different data shapes, and getting the stages to compose is most of the work. It exists because the methods are published faster than they are implemented, so a library that tracks them shortens the distance from paper to a trained model.
Technology Graph
How TRL connects to the rest of the stack.
Applied to
- DPO technique Techniques TRL used for DPO High ✓ source
- Fine-tuning technique Techniques TRL used for Fine-tuning High ✓ source
- RLHF technique Techniques TRL used for RLHF High ✓ source