Softmax
Turns a vector of scores into a probability distribution.
Aperçu
Softmax exponentiates each score and divides by the sum, producing values between zero and one that add up to one. It solves the problem that raw scores are unbounded and unnormalised, so they cannot be compared as a weighting. It exists wherever a choice has to be made from scores, which in a transformer is the attention weights and the output distribution over the vocabulary.
Graphe des technologies
Comment Softmax se rattache au reste de la pile.