Tokenizer
Component that converts text to tokens and back.
Aperçu
The tokenizer applies the vocabulary learned by a tokenisation scheme, splitting input text into token ids and joining ids back into text. It solves the interface problem between raw strings and the integer sequences a model consumes. It exists as a separate artefact from the weights, which is why a checkpoint is only usable alongside the tokenizer it was trained with.
Graphe des technologies
Comment Tokenizer se rattache au reste de la pile.
Outils
Logiciels qui implémentent Tokenizer ou s'y intègrent.