BERT
仅编码器 Transformer，采用掩码语言建模进行预训练。
概览
BERT 是一种仅含编码器的 Transformer，通过遮蔽句子中的单词并学习恢复它们来进行预训练，因此它构建的是丰富的文本表示，而非生成新文本。它解决了早期模型需要为每个下游理解任务单独设计架构的问题。它之所以存在，是因为双向上下文对理解至关重要：知道空缺两侧的单词对其含义的约束远大于仅知道左侧。它至今仍是编码器式语言模型的参照基准。
技术图谱
BERT 如何与技术栈的其他部分相连。
使用与依赖
- Transformer architecture Algorithms BERT uses Transformer High ✓ source
- BGE model Embedding Models BGE based on BERT High ✓ source
- Cross-Encoder concept Techniques Cross-Encoder based on BERT High ✓ source
- E5 model Embedding Models E5 based on BERT High ✓ source
- GTE model Embedding Models GTE based on BERT High ✓ source
- Instructor model Embedding Models Instructor based on BERT High ✓ source
- Reranking technique Techniques Reranking based on BERT High ✓ source
- Sentence Transformers framework Frameworks Sentence Transformers based on BERT High ✓ source
- Transformer architecture Algorithms BERT based on Transformer High ✓ source
应用于
- Distillation technique Techniques Distillation used for BERT High ✓ source