Psychologist Frank Rosenblatt built the first neural networks in 1957/8. Today it is trivial to build a neural network using software, but Rosenblatt built a neural network using analogue hardware.
The perceptron could tell the difference between triangles, circles and and squares, or between different letters of the alphabet. Most critically, it could get better at doing so as it was given feedback on whether it was correct or incorrect about previous predictions. In this sense, it could 'learn'.
A hardware implementation of a neural network
The input to the perceptron was an array of 20 x 20 grid of light sensitive resistors
And the weights/bias values of the neural network in the primitive neural network were stored in racks of cylindrical objects each consisting of an electrical motor and potentiometer (rotary resistor).