From machine learning to the Transformer

Learning behaviour from examples rather than writing rules for it.

Layers of weighted sums with nonlinearities, so the model can represent interactions a linear model cannot.

Enough layers that the network learns its own features instead of being handed them.

Letting every position look at every other one, which removes the bottleneck of compressing a sequence into a single state.

Attention without recurrence: parallel training over long sequences, and the architecture everything later in this graph is built on.