Evaluation
How model quality is measured: benchmarks, leaderboards, human preference, and the ways each can be gamed. Necessary context for reading any published number.
6 technologies
- LLM-as-a-Judge technique Using a capable model to score another model's output.
- LM Evaluation Harness tool Standard harness for running language model benchmarks.
- Ragas tool Evaluation library that scores retrieval and answer quality separately.
- LangSmith tool Tracing and evaluation for model applications, showing each step of a run.
- DeepEval library Open-source framework of tests for model applications.
- TruLens library Instrumentation and feedback functions for evaluating model applications.