Learning behaviour from examples rather than writing rules for it.
Technology path
From machine learning to the Transformer
The conceptual ladder. Each rung exists because the one below it was not enough for the problems people actually had.
5 steps · curated, not generated
-
-
Layers of weighted sums with nonlinearities, so the model can represent interactions a linear model cannot.
-
Enough layers that the network learns its own features instead of being handed them.
-
Letting every position look at every other one, which removes the bottleneck of compressing a sequence into a single state.
-
Attention without recurrence: parallel training over long sequences, and the architecture everything later in this graph is built on.