Red Teaming
Adversarially probing a model for harmful or unsafe behaviour.
Overview
Red teaming is the deliberate attempt to make a model produce output it should not, using prompt injection, roleplay framing and other adversarial techniques. It solves the problem that a model which refuses obvious harmful requests may still comply when the request is disguised. It exists because the failure is found by looking for it: a model cannot be assessed for unsafe behaviour by asking it politely, and the findings are what safety training is then measured against.
Technology Graph
How Red Teaming connects to the rest of the stack.
Uses and depends on
- Prompt Engineering technique Prompting Red Teaming uses Prompt Engineering High ✓ source
Applied to
- Alignment technique Safety & Alignment Red Teaming used for Alignment High ✓ source
- Guardrails technique Safety & Alignment Guardrails used for Red Teaming Medium ✓ source
- Jailbreak technique Safety & Alignment Jailbreak used for Red Teaming High ✓ source