BERT
Encoder-only Transformer pre-trained with masked language modelling.
Overview
BERT is an encoder-only Transformer pre-trained by masking words in a sentence and learning to recover them, so it builds a rich representation of text rather than generating new text. It solves the problem that earlier models required a separate architecture for every downstream understanding task. It exists because bidirectional context matters for understanding: knowing the words on both sides of a gap constrains its meaning far more than knowing only the left side. It remains the reference point for encoder-style language models.
Technology Graph
How BERT connects to the rest of the stack.
Uses and depends on
- Transformer architecture Algorithms BERT uses Transformer High ✓ source
- BGE model Embedding Models BGE based on BERT High ✓ source
- Cross-Encoder concept Techniques Cross-Encoder based on BERT High ✓ source
- E5 model Embedding Models E5 based on BERT High ✓ source
- GTE model Embedding Models GTE based on BERT High ✓ source
- Instructor model Embedding Models Instructor based on BERT High ✓ source
- Reranking technique Techniques Reranking based on BERT High ✓ source
- Sentence Transformers framework Frameworks Sentence Transformers based on BERT High ✓ source
- Transformer architecture Algorithms BERT based on Transformer High ✓ source
Applied to
- Distillation technique Techniques Distillation used for BERT High ✓ source