Transformer
Attention-based neural architecture that replaced recurrence with self-attention.
Overview
The Transformer is a neural network architecture that processes a whole sequence at once using self-attention instead of stepping through it one token at a time. It solves the problem that recurrent networks could not parallelise training and struggled to relate distant positions in a sequence. It exists because attention alone turned out to be sufficient for translation quality, and because removing recurrence made training scale across many accelerators. Almost every large language model in use today is a Transformer variant, which makes it the single most load-bearing node in this graph.
Technology Graph
How Transformer connects to the rest of the stack.
Uses and depends on
- Attention concept Algorithms Transformer based on Attention High ✓ source
- BERT model Models BERT based on Transformer High ✓ source
- CLIP model Multimodal CLIP based on Transformer High ✓ source
- DeepSeek model Models DeepSeek based on Transformer High ✓ source
- GPT model Models GPT based on Transformer High ✓ source
- Gemma model Models Gemma based on Transformer High ✓ source
- LLM concept Models LLM based on Transformer High ✓ source
- Llama model Models Llama based on Transformer High ✓ source
- Mistral model Models Mistral based on Transformer High ✓ source
- Multimodal AI application Multimodal Multimodal AI based on Transformer High ✓ source
- Phi model Models Phi based on Transformer High ✓ source
- Pretraining technique Techniques Pretraining based on Transformer High ✓ source
- Qwen model Models Qwen based on Transformer High ✓ source
- Scaling Laws concept Algorithms Scaling Laws based on Transformer High ✓ source
- Speech Recognition application Applications Speech Recognition based on Transformer High ✓ source
- ViT architecture Multimodal ViT based on Transformer High ✓ source
- Whisper model Multimodal Whisper based on Transformer High ✓ source
Used by
- BERT model Models Transformer used by BERT High ✓ source
- GPT model Models Transformer used by GPT High ✓ source
- Hugging Face Transformers library LLM Frameworks Transformer used by Hugging Face Transformers High ✓ source
Applied to
- Layer Normalization concept Algorithms Layer Normalization used for Transformer High ✓ source
- RoPE technique Techniques RoPE used for Transformer High ✓ source
- LLM concept Models Transformer enables LLM High ✓ source
Classification
- Feed-Forward Network concept Algorithms Feed-Forward Network part of Transformer High ✓ source
- Multi-Head Attention concept Algorithms Multi-Head Attention part of Transformer High ✓ source
- Positional Encoding concept Algorithms Positional Encoding part of Transformer High ✓ source
- Residual Connection concept Algorithms Residual Connection part of Transformer High ✓ source
- Self-Attention concept Algorithms Self-Attention part of Transformer High ✓ source
Alternatives
Comparisons
Side-by-side breakdowns with the alternatives above.