Tokenizer
Component that converts text to tokens and back.
Overview
The tokenizer applies the vocabulary learned by a tokenisation scheme, splitting input text into token ids and joining ids back into text. It solves the interface problem between raw strings and the integer sequences a model consumes. It exists as a separate artefact from the weights, which is why a checkpoint is only usable alongside the tokenizer it was trained with.
Technology Graph
How Tokenizer connects to the rest of the stack.
Tools
Software that implements or integrates with Tokenizer.