Tokenizer
Component that converts text to tokens and back.
Visão geral
The tokenizer applies the vocabulary learned by a tokenisation scheme, splitting input text into token ids and joining ids back into text. It solves the interface problem between raw strings and the integer sequences a model consumes. It exists as a separate artefact from the weights, which is why a checkpoint is only usable alongside the tokenizer it was trained with.
Grafo de tecnologias
Como Tokenizer se liga ao resto do stack.
Ferramentas
Software que implementa ou se integra com Tokenizer.