Tokenizer
Component that converts text to tokens and back.
Resumen
The tokenizer applies the vocabulary learned by a tokenisation scheme, splitting input text into token ids and joining ids back into text. It solves the interface problem between raw strings and the integer sequences a model consumes. It exists as a separate artefact from the weights, which is why a checkpoint is only usable alongside the tokenizer it was trained with.
Grafo de tecnologías
Cómo se conecta Tokenizer con el resto del stack.
Herramientas
Software que implementa o se integra con Tokenizer.