Universal Approximation Theorem states {that a} neural community with a single hidden layer and a nonlinear activation perform can approximate any steady perform.
Sensible points apart, such that the variety of neurons on this hidden layer would develop enormously giant, we don’t want different community architectures. A easy feed-forward neural community may do the trick.
It’s difficult to estimate what number of community architectures have been developed.
Whenever you open the favored AI mannequin platform Hugging Face right this moment, one can find a couple of million pretrained fashions. Relying on the duty, you’ll use completely different architectures, for instance transformers for pure language processing and convolutional networks for picture classification.
So, why will we want so many Neural Network architectures?
On this put up, I would like provide a solution to this query from a physics perspective. It’s the construction within the information that conjures up novel neural community architectures.
Symmetry and invariance
Physicists love symmetry. The basic legal guidelines of physics make use of symmetries, reminiscent of the truth that the movement of a particle could be described by the identical equations, no matter the place it finds itself in time and house.
Symmetry at all times implies invariance with respect to some transformation. These ice crystals are an instance of translational invariance. The smaller buildings look the identical, no matter the place they seem within the bigger context.
Exploiting symmetries: convolutional neural community
In the event you already know {that a} sure symmetry persists in your information, you’ll be able to exploit this reality to simplify your neural community structure.
Let’s clarify this with the instance of picture classification. The panel reveals three scenes together with a goldfish. It could possibly present up in any location inside the picture, however the picture ought to at all times be labeled as goldfish.
A feed-forward neural community may definitely obtain this, given enough coaching information.
This community structure requires a flattened enter picture. Weights are then assigned between every enter layer neuron (representing one pixel within the picture) and every hidden layer neuron. Additionally, weights are assigned between every neuron within the hidden and the output layer.
Together with this structure, the panel reveals a “flattened” model of the three goldfish photos from above. Do they nonetheless look alike to you?
By flattening the picture, now we have incurred two issues:
- Pictures that include an analogous object don’t look alike as soon as they’re flattened,
- For prime-resolution photos, we might want to practice plenty of weights connecting the enter layer and the hidden layer.
Convolutional networks, then again, work with kernels. Kernel sizes sometimes vary between 3 and seven pixels, and the kernel parameters are learnable in coaching.
The kernel is utilized like a raster to the picture. A convolutional layer could have a couple of kernel, permitting every kernel to give attention to completely different points of the picture.
For instance, one kernel would possibly decide up on horizontal traces within the picture, whereas one other would possibly decide up on convex curves.
Convolutional neural networks protect the order of pixels and are nice to study localized buildings. The convolutional layers could be nested to create deep layers. Together with pooling layers, high-level options could be discovered.
The ensuing networks are significantly smaller than in case you would use a fully-connected neural community. A convolutional layer solely requires kernel_size x kernel_size x n_kernel trainable parameters.
It can save you reminiscence and computational price range, all by exploiting the truth that your object could also be situated wherever inside your picture!
Extra superior deep studying architectures that exploit symmetries are Graph Neural Networks and physics-informed neural networks.
Abstract
Convolutional neural networks work nice with photos as a result of they protect the native data in your picture. As a substitute of flattening all of the pixels, rendering the picture meaningless, kernels with learnable parameters decide up on native options.
