Transformer
Attention-based neural architecture that replaced recurrence with self-attention.
Aperçu
The Transformer is a neural network architecture that processes a whole sequence at once using self-attention instead of stepping through it one token at a time. It solves the problem that recurrent networks could not parallelise training and struggled to relate distant positions in a sequence. It exists because attention alone turned out to be sufficient for translation quality, and because removing recurrence made training scale across many accelerators. Almost every large language model in use today is a Transformer variant, which makes it the single most load-bearing node in this graph.
Graphe des technologies
Comment Transformer se rattache au reste de la pile.
Utilise et dépend de
- Attention concept Algorithms Transformer based on Attention High ✓ source
- BERT model Models BERT based on Transformer High ✓ source
- CLIP model Multimodal CLIP based on Transformer High ✓ source
- DeepSeek model Models DeepSeek based on Transformer High ✓ source
- GPT model Models GPT based on Transformer High ✓ source
- Gemma model Models Gemma based on Transformer High ✓ source
- LLM concept Models LLM based on Transformer High ✓ source
- Llama model Models Llama based on Transformer High ✓ source
- Mistral model Models Mistral based on Transformer High ✓ source
- Multimodal AI application Multimodal Multimodal AI based on Transformer High ✓ source
- Phi model Models Phi based on Transformer High ✓ source
- Pretraining technique Techniques Pretraining based on Transformer High ✓ source
- Qwen model Models Qwen based on Transformer High ✓ source
- Scaling Laws concept Algorithms Scaling Laws based on Transformer High ✓ source
- Speech Recognition application Applications Speech Recognition based on Transformer High ✓ source
- ViT architecture Multimodal ViT based on Transformer High ✓ source
- Whisper model Multimodal Whisper based on Transformer High ✓ source
Utilisé par
- BERT model Models Transformer used by BERT High ✓ source
- GPT model Models Transformer used by GPT High ✓ source
- Hugging Face Transformers library LLM Frameworks Transformer used by Hugging Face Transformers High ✓ source
Appliqué à
- Layer Normalization concept Algorithms Layer Normalization used for Transformer High ✓ source
- RoPE technique Techniques RoPE used for Transformer High ✓ source
- LLM concept Models Transformer enables LLM High ✓ source
Classification
- Feed-Forward Network concept Algorithms Feed-Forward Network part of Transformer High ✓ source
- Multi-Head Attention concept Algorithms Multi-Head Attention part of Transformer High ✓ source
- Positional Encoding concept Algorithms Positional Encoding part of Transformer High ✓ source
- Residual Connection concept Algorithms Residual Connection part of Transformer High ✓ source
- Self-Attention concept Algorithms Self-Attention part of Transformer High ✓ source
Alternatives
Comparaisons
Comparaisons détaillées avec les alternatives ci-dessus.