# Deep learning / Deep neural nets for mathematician

I am interested in finding out the math ideas behind the technologies that are under the umbrella of "Deep Learning" or "Deep neural nets".

Most of the papers/books that are often quoted in papers/online as references are not written in a very math-friendly manner. I am specifically referring to the fact that this field is highly interdisciplinary, and the language used (e.g. 'levels', 'stacking networks') are not standard mathematical terminology, but rather very specialized terms.

So I am writing this post to find out if there exists a book or review article written for pure mathematicians about the core mathematical ideas of the whole deep-learning thing.

My hope is that is there is a reference that follows (sort of ) the theorem-lemma-proof format or at least tries to where ever possible, or at least gives some rigorous definitions so that I can make sense of the terminology.

Thank you.

• This is more for physicists but nevertheless looks like it should be pretty insightful: arxiv.org/abs/1410.3831 Apr 9 '15 at 14:36
• Do you mean to specify deep learning (the training of neural networks with multiple hidden layers, primarily developed in the last 9 years) as opposed to more standard techniques in machine learning including things like SVMs, universal representability, and the Vapnik–Chervonenkis dimension? Apr 9 '15 at 17:10
• I guess a bit of both, but my main motivation has been to see what the main idea of the new stuff is. I guess I would appreciate good references for both (new and old) from mathematical point of view.
– aa12
Apr 9 '15 at 20:19
• Also, one of the challenges to a formal understanding of deep learning methods is that the performance of a method depends both on the model ("architecture") and the parameter estimation algorithm ("training", which is often heuristic). Many engineering-oriented treatments conflate these two elements in ways that can be confusing. Apr 10 '15 at 3:38
• Have a look at this book: iro.umontreal.ca/~bengioy/dlbook --- that book is not so mathematical, but at least will show several kinds of the math that shows up in "learning" (probability, statistics, functional analysis, graph theory, optimization, etc.) Apr 12 '15 at 15:56

Update

The Coursera course I recommended long ago has now gone offline, although you can find links to the slides and videos on Hinton's home page. In any case, the field has continued to advance dramatically and there are new results and more up-to-date expository work; see any of the more recent answers.

For what it's worth, in the six years since I wrote this answer, the most fruitful point of view in my own work has been to focus on the high-dimensional geometry of neural networks. There are a lot of interesting sights to see in the wilds of a world with thousands or millions of dimensions.

If you have time, I highly recommend this Coursera course.

The videos are available for free and are truly excellent. The teacher is Geoffrey Hinton, who is one of the main players in the area, and he does an excellent job of providing both clear definitions and useful intuition.

In general, I wouldn't expect to see perfect theorem-lemma-proof exposition of deep learning anywhere, simply because the math hasn't caught up to real-world practice. More typical is a clean analysis of an idealized system, which is then related to a real system by a heuristic argument. In other words, this is an area that could use attention from mathematicians!

• The link is not functioning. Apr 1 '20 at 4:59
• Thanks for pointing out the link doesn't work. I've updated the answer. I still would recommend reading anything by Hinton, though! Jun 25 at 11:22
• still not working. Jun 25 at 11:34
• Added a link to a place where slides and videos seem preserved. Jun 25 at 12:11

Tommy Poggio et al's s MIT course seems great.

• The poster asked about math specific to deep learning, not to general machine learning / statistical learning theory. So this does not really answer the question I think. Apr 9 '15 at 22:19
• @Suvrit He asked for a reasonably jargon free rigorous explanation of the field, and this is it, I think. Apr 9 '15 at 22:49
• Yes, but a description of "Deep Learning" not of the field of machine learning --- but given the OPs comments, it seems that he/she may not be sure what they want. Apr 10 '15 at 2:30
• @Suvrit: The last two lectures are specifically on deep learning, and they provide a lot more connections to mathematics than most treatments of deep learning. Apr 12 '15 at 15:13

Chris Olah has a great blog post on how topology relates to machine learning ("machine learning untangles highly kneaded spaces").

I will let him summarize:

While it is challenging to understand the behavior of deep neural networks in general, it turns out to be much easier to explore low-dimensional deep neural networks – networks that only have a few neurons in each layer. In fact, we can create visualizations to completely understand the behavior and training of such networks. This perspective will allow us to gain deeper intuition about the behavior of neural networks and observe a connection linking neural networks to an area of mathematics called topology.

A number of interesting things follow from this, including fundamental lower-bounds on the complexity of a neural network capable of classifying certain datasets.

His blog also has posts on other specific types of deep neural networks such as "convolutional neural networks", but I haven't read those.

I have a blog post which discusses some of the connections between deep learning and advanced theoretical physics such as spin funnels and the renormalization group

http://charlesmartin14.wordpress.com/2015/03/25/why-does-deep-learning-work/

http://charlesmartin14.wordpress.com/2015/04/01/why-deep-learning-works-ii-the-renormalization-group/

Recently uploaded paper in Arxiv (1512.06293). This paper formalize concepts and proves them. And few others that you might check out:

1. J. Bruna and S. Mallat, “Invariant Scattering Convolution Networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1872–1886, Aug. 2013.

• for Covnet
2. A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, “The Loss Surfaces of Multilayer Networks,” arXiv:1412.0233 [cs], Nov. 2014.

• Spin Glasses Theory
• for Deep MLP and Convnet
3. P. Mehta and D. J. Schwab, “An exact mapping between the Variational Renormalization Group and Deep Learning,” arXiv:1410.3831 [cond-mat, stat], Oct. 2014.

• for Restricted Boltzmann Machine, Deep Belief Network and Deep Boltzmann Machine.

Since this question got bumped up to the front page somehow, I'm taking the liberty to suggest a partial introduction to the "Math of Deep Learning" given in the following article: The Modern Mathematics of Deep Learning.

Please have a look at Jason Morton's work on Mathematics of Deep Learning. It is quite mathematically rigorous if that's what is you are asking for.

Specifically, this paper

Note: The following is an answer to this post but everything I posted there applies equally and fully here.

Let me also comment shortly, that I also found the theory of DNNs difficult to enter since a clear "mathematically formal" point of access is now always clear in broader machine learning literature. Nevertheless, I found these helpful.

Shallow Feedforward Networks and Deep Convolutional Networks

I would suggest some Harmonic/Fourier analysis, some constructive approximation theory, and their intersection (esp.: Besov Spaces). This is because, many of the quantitative approximation theorems for shallow (1-hidden layer) feedforward networks are derived via such methods. Relevant (contemporary) papers for such methods include:

Deep Feedforward Networks and Optimal Rates

Otherwise, for deep feed-forward networks some of the more insightful approximation-theoretic results rely on Vapnik-Chervonekis Theory. These are then typically used to derive "optimal approximation rates"; see especially these papers:

Non-Euclidean Input/Output Spaces and Topological Embeddings

These results typically rely on results of a more topological flavor. I would Von Mill's book and of course basic general topology textbooks like Munkres' classic. The only universal approximation theorems I know of in this context are:

Recurrent Structures and Reservoir Computers

If you're looking for something a bit more "dynamic" in nature, then I would recommend brushing up on your functional analysis, measure theory, and sequences in Banach spaces. The first of these papers makes extensive use of ideas surrounding Rademacher Complexity and there are deep connections to the theory of dynamical systems.

I mention here also the developing connections between learning dynamics and rough path theory. See:

Qualitative Approximation by Shallow Feedforward Networks "Classical Style"

Let me mention that, classical (qualitative) universal approximation results are based on the Stone-Weierstraß theorem from approximation theory. Some results rely on the theory of LF-Spaces which are a class of Locally-Convex spaces with a particularly "category-theoretic$$\cap$$functional-analytic flavor". For modern formulations of the result in rather general contexts, see:

The last of these references needs only a bit of background in topological groups.

Memory Capacity/ Interpolation Capabilities These results have a variety of backgrounds. The latter of these results draws from the Chow-Rashevskii Theorem and control theory.

Impossibility Theorems Let me briefly round off this post with the following interesting results. The pre-requisits for these papers are a typical background; nonetheless, their results are fascinating.

I worked with neural nets in the 80's and haven't kept up with the literatue, so I don't have any links. However, I came away with the conclusion that training neural nets is essentially training a digital computer to behave like an analog computer. So, examining the mathematics pertaining to analog circuits might give some insight in this area.

• I find this interesting. Is it meant to be kind of a handwaving analogy or an ansatz that can be made rigorous? Feb 14 '16 at 21:09
• @DelioMugnolo, When I experimented with neural nets (as a hobby), I saw the analogy to bridge networks---plug in an unknown resistor (say) and read its value on a meter. The pieces of the bridge would be somewhat like the nodes of a neural net. It is only my take on the process. Feb 15 '16 at 7:23
• @DelioMugnolo, additional observation---D-WAVE's quantum computer is analog. Feb 15 '16 at 7:35
• Thanks. Do D-Wave computers have anything to do with neural nets? Feb 15 '16 at 8:06

I just found this paper-https://arxiv.org/pdf/1801.05894.pdf, which introduces deep learning in a mathematically sound manner, specially for computations of backpropagation etc. As a mathematician who worked in machine learning but not deep learning, I've noticed that the tools that're often needed for machine learning are linear algebra (with a bit of functional analysis), probability and optimization.

Apart from the above, there're also some literature using differential geometry and triangle meshes, e.g. https://arxiv.org/pdf/1611.08097.pdf, although I'm not quite sure whether they're being used in industrial applications. My suspicion comes from the fact that in general most of machine learning that uses differential geometry, e.g. manifold learning, is mostly useless outside academia, hence in real world, as real life observations don't satisfy smooth manifold assumption. However, I'd be glad, as a mathematician, to be proven wrong, in the sense if you could point out some true industrial use of geometric machine learning/geometric deep learning to solve realistic problems.

There's an ongoing course taught by Elchanan Mossel at MIT that you might find helpful. It really focuses on the things we can actually prove about deep learning, which may be mathematically appealing to you. The homepage is here.

The most interesting work that I've seen that unifies multiple different areas of mathematics to understand deep learning is Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges.

I'd say that deep learning (from a mathematician's perspective) is a HOT MESS. People are jumping through hoops trying to increase accuracy (sometimes just by decimals) introducing all sorts of stuff to models. They are like mad scientists, messing those poor neural networks.

Though the architecture of deep neural networks is deceptively simple, their structural composition just cannot explain everything. They depend on gazillion parameters, and a gazillion functions. Their dynamics just escapes traditional chaos theory. Bringing to the table stuff like functional analysis just doesn't cut it. Datasets are huge and unstructured. They somehow fail to hold any particular topological property (recall that everything in computers is discrete) so being mathematically rigorous with DNN calls for a bad migraine or will yield a very vague theory (see for example K. Weihrauch and computable analysis).

However, people like Yann LeCun and his energy based approach are onto something (my humble opinion) and there is a whole research field called explainability theory (kind of like psychology of DNN) where math plays an important role in developing a sort of psychiatry for Deep Neural Networks.

Eldad Haber at the University of British Columbia is exploring connections between neural networks, and dynamical systems: