Transformer
基于注意力的神经架构，用自注意力取代了循环。
概览
Transformer 是一种神经网络架构，它通过自注意力机制一次性处理整个序列，而不是逐个 token 依次推进。它解决了循环网络无法并行化训练、且难以关联序列中相距较远位置的问题。它之所以存在，是因为事实证明仅靠注意力就足以达到翻译质量，并且因为去除循环结构使训练得以在众多加速器上扩展。如今几乎每一个在用的大语言模型都是 Transformer 的变体，这使它成为本图谱中承载作用最大的单一节点。
技术图谱
Transformer 如何与技术栈的其他部分相连。
使用与依赖
- Attention concept Algorithms Transformer based on Attention High ✓ source
- BERT model Models BERT based on Transformer High ✓ source
- CLIP model Multimodal CLIP based on Transformer High ✓ source
- DeepSeek model Models DeepSeek based on Transformer High ✓ source
- GPT model Models GPT based on Transformer High ✓ source
- Gemma model Models Gemma based on Transformer High ✓ source
- LLM concept Models LLM based on Transformer High ✓ source
- Llama model Models Llama based on Transformer High ✓ source
- Mistral model Models Mistral based on Transformer High ✓ source
- Multimodal AI application Multimodal Multimodal AI based on Transformer High ✓ source
- Phi model Models Phi based on Transformer High ✓ source
- Pretraining technique Techniques Pretraining based on Transformer High ✓ source
- Qwen model Models Qwen based on Transformer High ✓ source
- Scaling Laws concept Algorithms Scaling Laws based on Transformer High ✓ source
- Speech Recognition application Applications Speech Recognition based on Transformer High ✓ source
- ViT architecture Multimodal ViT based on Transformer High ✓ source
- Whisper model Multimodal Whisper based on Transformer High ✓ source
被用于
- BERT model Models Transformer used by BERT High ✓ source
- GPT model Models Transformer used by GPT High ✓ source
- Hugging Face Transformers library LLM Frameworks Transformer used by Hugging Face Transformers High ✓ source
应用于
- Layer Normalization concept Algorithms Layer Normalization used for Transformer High ✓ source
- RoPE technique Techniques RoPE used for Transformer High ✓ source
- LLM concept Models Transformer enables LLM High ✓ source
Classification
- Feed-Forward Network concept Algorithms Feed-Forward Network part of Transformer High ✓ source
- Multi-Head Attention concept Algorithms Multi-Head Attention part of Transformer High ✓ source
- Positional Encoding concept Algorithms Positional Encoding part of Transformer High ✓ source
- Residual Connection concept Algorithms Residual Connection part of Transformer High ✓ source
- Self-Attention concept Algorithms Self-Attention part of Transformer High ✓ source
替代方案
对比
与上述替代方案的逐项对比。