Tokenizer
Component that converts text to tokens and back.
Überblick
The tokenizer applies the vocabulary learned by a tokenisation scheme, splitting input text into token ids and joining ids back into text. It solves the interface problem between raw strings and the integer sequences a model consumes. It exists as a separate artefact from the weights, which is why a checkpoint is only usable alongside the tokenizer it was trained with.
Technologiegraph
Wie Tokenizer mit dem übrigen Stack zusammenhängt.
Werkzeuge
Software, die Tokenizer implementiert oder integriert.