Tokenization
Splitting text into the subword units a language model actually reads.
Overview
Tokenization splits text into the subword units a model consumes, usually chosen so common words are one token and rare words are assembled from pieces. It solves the vocabulary problem: a word-level vocabulary cannot cover every name and inflection, and a character-level one makes sequences unnecessarily long. It exists because it sits between raw text and the model, and its choices shape context limits, pricing and how well a model handles unusual strings.
Technology Graph
How Tokenization connects to the rest of the stack.