Tokenization

Splitting text into the subword units a language model actually reads.

technique Técnicas 3 relationships

Visão geral

Tokenization splits text into the subword units a model consumes, usually chosen so common words are one token and rare words are assembled from pieces. It solves the vocabulary problem: a word-level vocabulary cannot cover every name and inflection, and a character-level one makes sequences unnecessarily long. It exists because it sits between raw text and the model, and its choices shape context limits, pricing and how well a model handles unusual strings.

Grafo de tecnologias

Como Tokenization se liga ao resto do stack.

Usado por

LLM concept Models Tokenization used by LLM High ✓ source

Aplicado a

BPE technique Techniques BPE used for Tokenization High ✓ source

Classification

Token concept Models Token part of Tokenization High ✓ source

Posição da tecnologia

AI
↓
Técnicas
↓
technique
↓
Tokenization

Tecnologias relacionadas

BPE technique
LLM concept
Token concept

Recursos oficiais

Ainda não há links oficiais registrados.

Última atualização: 2026-10-08
Última verificação: 2026-10-08

Fontes citadas

Cada relação nesta página inclui a sua fonte.

arxiv.org 2 claims
arxiv.org 1 claim