Softmax
Turns a vector of scores into a probability distribution.
Overview
Softmax exponentiates each score and divides by the sum, producing values between zero and one that add up to one. It solves the problem that raw scores are unbounded and unnormalised, so they cannot be compared as a weighting. It exists wherever a choice has to be made from scores, which in a transformer is the attention weights and the output distribution over the vocabulary.
Technology Graph
How Softmax connects to the rest of the stack.