Algorithms
The methods that make models work: attention, backpropagation, gradient descent, tokenisation and the rest. Each entry explains the problem it solves and what it replaced.
30 technologies
- Transformer architecture Attention-based neural architecture that replaced recurrence with self-attention.
- Attention concept Mechanism that weights each part of an input by its relevance to every other part.
- Backpropagation technique Computing gradients through a network by applying the chain rule backwards.
- Self-Attention concept Weighting every position by every other position in the same sequence.
- Gradient Descent technique Updating weights in the direction that reduces the loss.
- Deep Learning concept Machine learning using many-layered neural networks to learn representations from raw data.
- Multi-Head Attention concept Running several attention operations in parallel over different projections.
- HNSW algorithm Graph index that finds approximate nearest neighbours in logarithmic time.
- Residual Connection concept Adding a layer's input to its output so gradients can pass through unchanged.
- Scaling Laws concept Empirical relationship between model size, data, compute and loss.
- Contrastive Learning technique Training by pulling matching pairs together and pushing others apart.
- Neural Network concept Layered composition of weighted sums and nonlinearities, trained by gradient descent.
- BF16 concept Sixteen-bit float that keeps FP32's exponent range for training stability.
- Learning Rate concept How far each gradient step moves the weights.
- Overfitting concept A model fitting its training data instead of the pattern behind it.
- Positional Encoding concept How a transformer, which has no inherent order, learns where tokens are.
- Feed-Forward Network concept The position-wise fully connected layer inside each transformer block.
- Softmax concept Turns a vector of scores into a probability distribution.
- Regularization concept Any constraint that keeps a model from fitting its training data too closely.
- Machine Learning concept Field of building systems that improve at a task from data rather than from explicit rules.
- FP8 concept Eight-bit floating point, used to train and serve models faster.
- Layer Normalization concept Normalising activations across features within each sample.
- CNN architecture Convolutional neural network, sharing weights across positions to exploit spatial structure.
- Vanishing Gradient concept Gradients shrinking toward zero as they propagate through many layers.