Deep learning / Deep neural nets for mathematician

Question

I am interested in finding out the math ideas behind the technologies that are under the umbrella of "Deep Learning" or "Deep neural nets".

Most of the papers/books that are often quoted in papers/online as references are not written in a very math-friendly manner. I am specifically referring to the fact that this field is highly interdisciplinary, and the language used (e.g. 'levels', 'stacking networks') are not standard mathematical terminology, but rather very specialized terms.

So I am writing this post to find out if there exists a book or review article written for pure mathematicians about the core mathematical ideas of the whole deep-learning thing.

My hope is that is there is a reference that follows (sort of ) the theorem-lemma-proof format or at least tries to where ever possible, or at least gives some rigorous definitions so that I can make sense of the terminology.

Thank you.

Answer 1

Update

The Coursera course I recommended long ago has now gone offline, although you can find links to the slides and videos on Hinton's home page. In any case, the field has continued to advance dramatically and there are new results and more up-to-date expository work; see any of the more recent answers.

For what it's worth, in the six years since I wrote this answer, the most fruitful point of view in my own work has been to focus on the high-dimensional geometry of neural networks. There are a lot of interesting sights to see in the wilds of a world with thousands or millions of dimensions.

Old answer

If you have time, I highly recommend this Coursera course.

The videos are available for free and are truly excellent. The teacher is Geoffrey Hinton, who is one of the main players in the area, and he does an excellent job of providing both clear definitions and useful intuition.

In general, I wouldn't expect to see perfect theorem-lemma-proof exposition of deep learning anywhere, simply because the math hasn't caught up to real-world practice. More typical is a clean analysis of an idealized system, which is then related to a real system by a heuristic argument. In other words, this is an area that could use attention from mathematicians!

Answer 2

Tommy Poggio et al's s MIT course seems great.

Answer 3

Chris Olah has a great blog post on how topology relates to machine learning ("machine learning untangles highly kneaded spaces").

I will let him summarize:

While it is challenging to understand the behavior of deep neural networks in general, it turns out to be much easier to explore low-dimensional deep neural networks – networks that only have a few neurons in each layer. In fact, we can create visualizations to completely understand the behavior and training of such networks. This perspective will allow us to gain deeper intuition about the behavior of neural networks and observe a connection linking neural networks to an area of mathematics called topology.

A number of interesting things follow from this, including fundamental lower-bounds on the complexity of a neural network capable of classifying certain datasets.

His blog also has posts on other specific types of deep neural networks such as "convolutional neural networks", but I haven't read those.

Answer 4

I have a blog post which discusses some of the connections between deep learning and advanced theoretical physics such as spin funnels and the renormalization group

https://calculatedcontent.com/2015/03/25/why-does-deep-learning-work/

https://calculatedcontent.com/2015/04/01/why-deep-learning-works-ii-the-renormalization-group/

Answer 5

Since this question got bumped up to the front page somehow, I'm taking the liberty to suggest a partial introduction to the "Math of Deep Learning" given in the following article: The Modern Mathematics of Deep Learning.

Answer 6

Recently uploaded paper in Arxiv (1512.06293). This paper formalize concepts and proves them. And few others that you might check out:

1. J. Bruna and S. Mallat, “Invariant Scattering Convolution Networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1872–1886, Aug. 2013.

• for Covnet

2. A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, “The Loss Surfaces of Multilayer Networks,” arXiv:1412.0233 [cs], Nov.

• Spin Glasses Theory
• for Deep MLP and Convnet

3. P. Mehta and D. J. Schwab, “An exact mapping between the Variational Renormalization Group and Deep Learning,” arXiv:1410.3831 [cond-mat, stat], Oct. 2014.

• for Restricted Boltzmann Machine, Deep Belief Network and Deep Boltzmann Machine.

Answer 7

I just found this paper-https://arxiv.org/pdf/1801.05894.pdf, which introduces deep learning in a mathematically sound manner, specially for computations of backpropagation etc. As a mathematician who worked in machine learning but not deep learning, I've noticed that the tools that're often needed for machine learning are linear algebra (with a bit of functional analysis), probability and optimization.

Apart from the above, there're also some literature using differential geometry and triangle meshes, e.g. https://arxiv.org/pdf/1611.08097.pdf, although I'm not quite sure whether they're being used in industrial applications. My suspicion comes from the fact that in general most of machine learning that uses differential geometry, e.g. manifold learning, is mostly useless outside academia, hence in real world, as real life observations don't satisfy smooth manifold assumption. However, I'd be glad, as a mathematician, to be proven wrong, in the sense if you could point out some true industrial use of geometric machine learning/geometric deep learning to solve realistic problems.

Answer 8

Please have a look at Jason Morton's work on Mathematics of Deep Learning. It is quite mathematically rigorous if that's what is you are asking for.

Link: http://www.jasonmorton.com/publications.html (Wayback Machine)

Specifically, this paper

Answer 9

Note: The following is an answer to this post but everything I posted there applies equally and fully here.

Let me also comment shortly, that I also found the theory of DNNs difficult to enter since a clear "mathematically formal" point of access is now always clear in broader machine learning literature. Nevertheless, I found these helpful.

---

---

---

Shallow Feedforward Networks and Deep Convolutional Networks

I would suggest some Harmonic/Fourier analysis, some constructive approximation theory, and their intersection (esp.: Besov Spaces). This is because, many of the quantitative approximation theorems for shallow (1-hidden layer) feedforward networks are derived via such methods. Relevant (contemporary) papers for such methods include:

• Approximation spaces of deep neural networks - Gribonval et al. 2021
• Approximation rates for neural networks with general activation functions - Siegel and Xu, 2021
• Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality - Taiji Suzuki This particular point carries over equally to (deep) convolutional networks:
• Universality of deep convolutional neural networks - Ding-Xuan Zhou 2020

---

Deep Feedforward Networks and Optimal Rates

Otherwise, for deep feed-forward networks some of the more insightful approximation-theoretic results rely on Vapnik-Chervonekis Theory. These are then typically used to derive "optimal approximation rates"; see especially these papers:

• Optimal approximation of continuous functions by very deep ReLU networks D. Yarotsky 2018
• The phase diagram of approximation rates for deep neural networks - Dmitry Yarotsky, Anton Zhevnerchuk - 2020

---

Non-Euclidean Input/Output Spaces and Topological Embeddings

These results typically rely on results of a more topological flavor. I would Van Mill's book and of course basic general topology textbooks like Munkres' classic. The only universal approximation theorems I know of in this context are:

• Non-Euclidean Universal Approximation, 2020
• NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representation, 2021

---

Recurrent Structures and Reservoir Computers

If you're looking for something a bit more "dynamic" in nature, then I would recommend brushing up on your functional analysis, measure theory, and sequences in Banach spaces. The first of these papers makes extensive use of ideas surrounding Rademacher Complexity and there are deep connections to the theory of dynamical systems.

• Risk Bounds for Reservoir Computing - Gonon, Grigoryeva, Ortega - 2020
• Differentiable reservoir computing - Grigoryeva, Ortega - 2019

I mention here also the developing connections between learning dynamics and rough path theory. See:

• Deep signature transforms - Kidger, Bonnier, Perez Arribas, Salvi, Lyons - 2019
• Discrete-Time Signatures and Randomness in Reservoir Computing - Cuchiero, Gonon, Grigoryeva, Ortega, Teichmann - 2021

---

Qualitative Approximation by Shallow Feedforward Networks "Classical Style"

Let me mention that, classical (qualitative) universal approximation results are based on the Stone-Weierstraß theorem from approximation theory. Some results rely on the theory of LF-Spaces which are a class of Locally-Convex spaces with a particularly "category-theoretic$\cap$functional-analytic flavor". For modern formulations of the result in rather general contexts, see:

• Stone–Weierstraß and extension theorems in the nonlocally convex case - Timofte, Timofte, Khan - 2018
• Stone-weierstraß theorems for group-valued functions - Galindo, Sanchis, 2004

The last of these references needs only a bit of background in topological groups.

---

Memory Capacity/ Interpolation Capabilities These results have a variety of backgrounds. The latter of these results draws from the Chow-Rashevskii Theorem and control theory.

• Small ReLu networks are powerful memorizers: A tight analysis of memorization capacity - Yun, Sra, Jadbabaie - 2019
• Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations - Vershynin - 2020
• Deep Neural Networks, Generic Universal Interpolation, and Controlled ODEs - Cuchiero, Larsson, Teichmann - 2020

---

Impossibility Theorems Let me briefly round off this post with the following interesting results. The pre-requisits for these papers are a typical background; nonetheless, their results are fascinating.

• Minimum Width for Universal Approximation - Park, Yun, Lee, Shin - 2021
• Deep, Skinny neural networks are not universal approximators - Johnson - 2018

Answer 10

There's an ongoing course taught by Elchanan Mossel at MIT that you might find helpful. It really focuses on the things we can actually prove about deep learning, which may be mathematically appealing to you. The homepage is here.

Answer 11

Eldad Haber at the University of British Columbia is exploring connections between neural networks, and dynamical systems:

• (2018) Deep Neural Networks Motivated by Partial Differential Equations

• (2019) AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks

Update: that a CNN or ConvNet (particularly ResNet) is like a PDE (input as initial condition, convolution with 3x3 kernel is like a differential operator, the parameters/weights are the coefficients) has been observed by several people, and it wouldn't be a surprise if Yann LeCun, who introduced and coined the term "convolutional neural nets", was aware of it. When people say "ResNet is an ODE", or to call it "Neural ODE", they really should have said PDE. This perspective sheds light on many (if not all) of the designs of CNN architecture.

More specifically, the ResNet is a system of (nonlinear) PDEs, or a PDE with matrix coefficients. A natural question would be if this turns out (i.e., after successful training) to be a hyperbolic system that models wave propagation. Other questions that would be interesting from a mathematical point of view include if the dynamical system is chaotic or (partially) integrable, and how to define these concepts properly in this context. Can it help us design better CNNs by, say, adding variable coefficients, other boundary conditions, and nonlinearities other than pointwise ReLU? Can we regard the "self-attention" in Transformer as a cubic term?

Another implication of this perspective is to regard the training of a neural net as an Optimal Control (or dynamical programming) problem.

Answer 12

I worked with neural nets in the 80's and haven't kept up with the literature, so I don't have any links. However, I came away with the conclusion that training neural nets is essentially training a digital computer to behave like an analog computer. So, examining the mathematics pertaining to analog circuits might give some insight in this area.

Answer 13

I'd say that deep learning (from a mathematician's perspective) is a HOT MESS. People are jumping through hoops trying to increase accuracy (sometimes just by decimals) introducing all sorts of stuff to models. They are like mad scientists, messing those poor neural networks.

Though the architecture of deep neural networks is deceptively simple, their structural composition just cannot explain everything. They depend on gazillion parameters, and a gazillion functions. Their dynamics just escapes traditional chaos theory. Bringing to the table stuff like functional analysis just doesn't cut it. Datasets are huge and unstructured. They somehow fail to hold any particular topological property (recall that everything in computers is discrete) so being mathematically rigorous with DNN calls for a bad migraine or will yield a very vague theory (see for example K. Weihrauch and computable analysis).

However, people like Yann LeCun and his energy based approach are onto something (my humble opinion) and there is a whole research field called explainability theory (kind of like psychology of DNN) where math plays an important role in developing a sort of psychiatry for Deep Neural Networks.

Answer 14

The original question was asked in 2015. So I believe it is appropriate to include surveys with mathematical flavor on more recent/advanced topics in neural networks and deep learning.

1. A Mathematical Introduction to Generative Adversarial Nets (GAN) - 2020

The survey paper rigorously formulate the problem that a Generative Adversarial Network (GAN) tries to solve as a min-max optimization problem (Theorem 2.3).

2. Neural Network Theory - 2022

A quite bit of analysis is involved in studying the Expressive Power of Neural Networks, especially when it comes to approximation theorems. The lecture notes above survey classical universal approximation results for shallow neural nets as well as more advanced topics including approximation results under the manifold hypothesis, benefits of depth to expressivity, and results on the VC dimension and the convexity of the set of functions implemented by feedforward neural networks.

(Another mathematical question pertaining to this topic which can be of interest to a combinatorialist is to count/bound the number of activation regions of a ReLU network as a measure of its complexity/expressiveness; see this paper for instance.)

3. Graph Representation Learning - 2020

After a review of older methods of node embedding (e.g. using random walks on graphs), the book presents a careful treatment of Graph Neural Networks (GNNs). The message passing framework in GNNs is rigorously defined; and it is discussed how different update and aggregation operations in message passing yield different types of GNNs (Graph Convolutional Networks, Graph Attention Networks, GraphSAGE etc.). Furthermore, there is a chapter discussing the mathematical motivation such as the graph Fourier transform and the Weisfeiler-Lehman algorithm.

4. Geometric Deep Learning: Going beyond Euclidean data - 2017

The article provides an overview of Geometric Deep Learning, an "umbrella term" for deep learning on "non-Euclidean structured domains". This goes beyond graph neural networks and involves many problems in which a geometric prior (e.g. symmetry or scale separation) is present such as when the data lives on a submanifold, or Convolutional Neural Networks (CNNs) in which a grid-like structure is exploited. A more comprehensive and recent treatment can be found in this book which brands geometric deep learning as an "attempt to apply the Erlangen Programme mindset to the domain of deep learning".

5. A mathematician's introduction to transformers and large language models - 2022
   Formal Algorithms for Transformers - 2022

No theorems are involved but these surveys on Transformers were rigorous enough for my taste. The first one explains the attention mechanism, and the second one precisely presents the algorithms for various components of a transformer (positional embedding, attention, layer normalization etc.). Both references are self-contained.

6. Large Language Models - 2023

The paper gives an introduction to Large Language Models (LLMs) "for mathematicians, physicists, and other scientists and readers who are mathematically knowledgeable but not necessarily expert in machine learning or artificial intelligence."