Tokenizer
Component that converts text to tokens and back.
Обзор
The tokenizer applies the vocabulary learned by a tokenisation scheme, splitting input text into token ids and joining ids back into text. It solves the interface problem between raw strings and the integer sequences a model consumes. It exists as a separate artefact from the weights, which is why a checkpoint is only usable alongside the tokenizer it was trained with.
Граф технологий
Как Tokenizer связана с остальным стеком.
Инструменты
ПО, которое реализует Tokenizer или интегрируется с ним.