Token
The unit of text a language model reads and writes.
Overview
A token is the unit a language model operates on, typically a word or a fragment of one. It solves the problem that characters are too fine-grained and words too coarse to form a fixed vocabulary. It exists as an implementation detail with consequences that reach the user: it sets what a model can spell, what it costs, how long a context is, and why a model may fail on a string it has never seen tokenised.
Technology Graph
How Token connects to the rest of the stack.
Classification
- LLM concept Models Token part of LLM High ✓ source
- Tokenization technique Techniques Token part of Tokenization High ✓ source