BPE
Tokenisation that builds a vocabulary by merging frequent character pairs.
Обзор
Byte Pair Encoding starts from individual characters and repeatedly merges the most frequent adjacent pair, so common words become one token and rare ones are assembled from parts. It solves the vocabulary problem: a word-level vocabulary cannot cover every name and inflection, and a character-level one makes sequences unnecessarily long. It exists as the scheme most language models still use, with its vocabulary size chosen rather than learned.
Граф технологий
Как BPE связана с остальным стеком.
Применяется к
- Tokenization technique Techniques BPE used for Tokenization High ✓ source
Инструменты
ПО, которое реализует BPE или интегрируется с ним.