A GPU is a processor with thousands of simple cores optimised for doing the same operation across large arrays at once. It solves the problem that training and serving a neural network is almost entirely matrix multiplication, which is exactly that pattern, while a CPU spends its few complex cores poorly on it. It exists as the reason deep learning became practical: the hardware already existed for graphics, and it happened to fit the maths.
A TPU is an application-specific chip designed for the matrix operations neural networks are made of, rather than a general-purpose processor adapted to them. It solves the same cost problem as a GPU but by removing the generality, which makes it more efficient per operation and less flexible. It exists because a workload this narrow can be served better by dedicated silicon, and it is reachable mainly through Google's cloud rather than as hardware you buy.