LLM-as-a-Judge
Using a capable model to score another model's output.
Overview
LLM-as-a-judge asks a strong model to rate or compare outputs, usually with a rubric and a reference. It solves the problem that open-ended generation has no correct answer to compare against, so the most important capability was the least measurable. It exists because the correlation with human preference turned out to be usable, and its known weaknesses - position bias, verbosity bias, self-preference - are why the rubric matters.
Technology Graph
How LLM-as-a-Judge connects to the rest of the stack.
Used by
- MT-Bench benchmark Benchmarks LLM-as-a-Judge used by MT-Bench High ✓ source
- RLAIF technique Safety & Alignment LLM-as-a-Judge used by RLAIF High ✓ source