Token
The unit of text a language model reads and writes.
Aperçu
A token is the unit a language model operates on, typically a word or a fragment of one. It solves the problem that characters are too fine-grained and words too coarse to form a fixed vocabulary. It exists as an implementation detail with consequences that reach the user: it sets what a model can spell, what it costs, how long a context is, and why a model may fail on a string it has never seen tokenised.
Graphe des technologies
Comment Token se rattache au reste de la pile.
Classification
- LLM concept Models Token part of LLM High ✓ source
- Tokenization technique Techniques Token part of Tokenization High ✓ source