Deep learning / Deep neural nets for mathematician I am interested in finding out the math ideas behind the technologies that are under the umbrella of "Deep Learning" or "Deep neural nets".
Most of the papers/books that are often quoted in papers/online as references are not written in a very math-friendly manner. I am specifically referring to the fact that this field is highly interdisciplinary, and the language used (e.g. 'levels', 'stacking networks') are not standard mathematical terminology, but rather very specialized terms.
So I am writing this post to find out if there exists a book or review article written for pure mathematicians about the core mathematical ideas of the whole deep-learning thing.
My hope is that is there is a reference that follows (sort of ) the theorem-lemma-proof format or at least tries to where ever possible, or at least gives some rigorous definitions so that I can make sense of the terminology.
Thank you.
 A: Recently uploaded paper in Arxiv (1512.06293). This paper formalize concepts and proves them. And few others that you might check out:

*

*J. Bruna and S. Mallat, “Invariant Scattering Convolution
Networks,” IEEE Transactions on Pattern Analysis and Machine
Intelligence, vol. 35, no. 8, pp. 1872–1886, Aug. 2013.


*

*for Covnet



*A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun,
“The Loss Surfaces of Multilayer Networks,” arXiv:1412.0233 [cs], Nov.

*


*

*Spin Glasses Theory

*for Deep MLP and Convnet



*P. Mehta and D. J. Schwab, “An exact mapping between the
Variational Renormalization Group and Deep Learning,” arXiv:1410.3831
[cond-mat, stat], Oct. 2014.


*

*for Restricted Boltzmann Machine, Deep Belief Network and Deep
Boltzmann Machine.

A: Since this question got bumped up to the front page somehow, I'm taking the liberty to suggest a partial introduction to the "Math of Deep Learning" given in the following article: The Modern Mathematics of Deep Learning.
A: Please have a look at Jason Morton's work on Mathematics of Deep Learning. It is quite mathematically rigorous if that's what is you are asking for.
Link: http://www.jasonmorton.com/publications.html (Wayback Machine)
Specifically, this paper
A: I just found this paper-https://arxiv.org/pdf/1801.05894.pdf, which introduces deep learning in a mathematically sound manner, specially for computations of backpropagation etc. As a mathematician who worked in machine learning but not deep learning, I've noticed that the tools that're often needed for machine learning are linear algebra (with a bit of functional analysis), probability and optimization. 
Apart from the above, there're also some literature using differential geometry and triangle meshes, e.g. https://arxiv.org/pdf/1611.08097.pdf, although I'm not quite sure whether they're being used in industrial applications. My suspicion comes from the fact that in general most of machine learning that uses differential geometry, e.g. manifold learning, is mostly useless outside academia, hence in real world, as real life observations don't satisfy smooth manifold assumption. However, I'd be glad, as a mathematician, to be proven wrong, in the sense if you could point out some true industrial use of geometric machine learning/geometric deep learning to solve realistic problems.
A: Note: The following is an answer to this post but everything I posted there applies equally and fully here.
Let me also comment shortly, that I also found the theory of DNNs difficult to enter since a clear "mathematically formal" point of access is now always clear in broader machine learning literature.  Nevertheless, I found these helpful.



Shallow Feedforward Networks and Deep Convolutional Networks
I would suggest some Harmonic/Fourier analysis, some constructive approximation theory, and their intersection (esp.: Besov Spaces).  This is because, many of the quantitative approximation theorems for shallow (1-hidden layer) feedforward networks are derived via such methods.  Relevant (contemporary) papers for such methods include:

*

*Approximation spaces of deep neural networks - Gribonval et al. 2021

*Approximation rates for neural networks with general activation functions - Siegel and Xu, 2021

*Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality - Taiji Suzuki
This particular point carries over equally to (deep) convolutional networks:

*Universality of deep convolutional neural networks - Ding-Xuan Zhou 2020

Deep Feedforward Networks and Optimal Rates
Otherwise, for deep feed-forward networks some of the more insightful approximation-theoretic results rely on Vapnik-Chervonekis Theory.  These are then typically used to derive "optimal approximation rates"; see especially these papers:

*

*Optimal approximation of continuous functions by very deep ReLU networks D. Yarotsky 2018

*The phase diagram of approximation rates for deep neural networks - Dmitry Yarotsky, Anton Zhevnerchuk - 2020

Non-Euclidean Input/Output Spaces and Topological Embeddings
These results typically rely on results of a more topological flavor.  I would Van Mill's book and of course basic general topology textbooks like Munkres' classic.  The only  universal approximation theorems I know of in this context are:

*

*Non-Euclidean Universal Approximation, 2020

*NEU: A Meta-Algorithm for Universal UAP-Invariant Feature Representation, 2021

Recurrent Structures and Reservoir Computers
If you're looking for something a bit more "dynamic" in nature, then I would recommend brushing up on your functional analysis, measure theory, and sequences in Banach spaces.  The first of these papers makes extensive use of ideas surrounding Rademacher Complexity and there are deep connections to the theory of dynamical systems.

*

*Risk Bounds for Reservoir Computing - Gonon, Grigoryeva, Ortega - 2020

*Differentiable reservoir computing - Grigoryeva, Ortega - 2019
I mention here also the developing connections between learning dynamics and rough path theory.  See:

*

*Deep signature transforms - Kidger, Bonnier, Perez Arribas, Salvi, Lyons - 2019

*Discrete-Time Signatures and Randomness in Reservoir Computing - Cuchiero, Gonon, Grigoryeva, Ortega, Teichmann - 2021

Qualitative Approximation by Shallow Feedforward Networks "Classical Style"
Let me mention that, classical (qualitative) universal approximation results are based on the Stone-Weierstraß theorem from approximation theory.  Some results rely on the theory of LF-Spaces which are a class of Locally-Convex spaces with a particularly "category-theoretic$\cap$functional-analytic flavor".  For modern formulations of the result in rather general contexts, see:

*

*Stone–Weierstraß and extension theorems in the nonlocally
convex case - Timofte, Timofte, Khan - 2018

*Stone-weierstraß theorems for group-valued functions - Galindo, Sanchis, 2004
The last of these references needs only a bit of background in topological groups.

Memory Capacity/ Interpolation Capabilities
These results have a variety of backgrounds.  The latter of these results draws from the Chow-Rashevskii Theorem and control theory.

*

*Small ReLu networks are powerful memorizers:  A tight analysis of memorization capacity - Yun, Sra, Jadbabaie - 2019

*Memory Capacity of Neural Networks with Threshold and Rectified Linear Unit Activations -  Vershynin - 2020

*Deep Neural Networks, Generic Universal Interpolation, and Controlled ODEs - Cuchiero, Larsson, Teichmann - 2020

Impossibility Theorems
Let me briefly round off this post with the following interesting results.  The pre-requisits for these papers are a typical background; nonetheless, their results are fascinating.

*

*Minimum Width for Universal Approximation - Park, Yun, Lee, Shin - 2021

*Deep, Skinny neural networks are not universal approximators - Johnson - 2018
A: I worked with neural nets in the 80's and haven't kept up with the literature, so I don't have any links. However, I came away with the conclusion that training neural nets is essentially training a digital computer to behave like an analog computer. So, examining the mathematics pertaining to analog circuits might give some insight in this area.
A: There's an ongoing course taught by Elchanan Mossel at MIT that you might find helpful. It really focuses on the things we can actually prove about deep learning, which may be mathematically appealing to you. The homepage is here.
A: Eldad Haber at the University of British Columbia is exploring connections between neural networks, and dynamical systems:

*

*(2018) Deep Neural Networks Motivated by Partial Differential Equations


*(2019) AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks
Update: that a CNN or ConvNet (particularly ResNet) is like a PDE (input as initial condition, convolution with 3x3 kernel is like a differential operator, the parameters/weights are the coefficients) has been observed by several people, and it wouldn't be a surprise if Yann LeCun, who introduced and coined the term "convolutional neural nets", was aware of it. When people say "ResNet is an ODE", or to call it "Neural ODE", they really should have said PDE. This perspective sheds light on many (if not all) of the designs of CNN architecture.
More specifically, the ResNet is a system of (nonlinear) PDEs, or a PDE with matrix coefficients. A natural question would be if this turns out (i.e., after successful training) to be a hyperbolic system that models wave propagation. Other questions that would be interesting from a mathematical point of view include if the dynamical system is chaotic or (partially) integrable, and how to define these concepts properly in this context. Can it help us design better CNNs by, say, adding variable coefficients, other boundary conditions, and nonlinearities other than pointwise ReLU? Can we regard the "self-attention" in Transformer as a cubic term?
Another implication of this perspective is to regard the training of a neural net as an Optimal Control (or dynamical programming) problem.
A: Update
The Coursera course I recommended long ago has now gone offline, although you can find links to the slides and videos on Hinton's home page. In any case, the field has continued to advance dramatically and there are new results and more up-to-date expository work; see any of the more recent answers.
For what it's worth, in the six years since I wrote this answer, the most fruitful point of view in my own work has been to focus on the high-dimensional geometry of neural networks. There are a lot of interesting sights to see in the wilds of a world with thousands or millions of dimensions.
Old answer
If you have time, I highly recommend this Coursera course.
The videos are available for free and are truly excellent. The teacher is Geoffrey Hinton, who is one of the main players in the area, and he does an excellent job of providing both clear definitions and useful intuition.
In general, I wouldn't expect to see perfect theorem-lemma-proof exposition of deep learning anywhere, simply because the math hasn't caught up to real-world practice. More typical is a clean analysis of an idealized system, which is then related to a real system by a heuristic argument. In other words, this is an area that could use attention from mathematicians!
A: I'd say that deep learning (from a mathematician's perspective) is a HOT MESS. People are jumping through hoops trying  to increase accuracy (sometimes just by decimals) introducing all sorts of stuff to models. They are like mad scientists, messing those poor neural networks. 
Though the architecture of deep neural networks is deceptively simple, their structural composition just cannot explain everything. They depend on gazillion parameters, and a gazillion functions. Their dynamics just escapes traditional chaos theory. Bringing to the table stuff like functional analysis just doesn't cut it. Datasets are huge and unstructured. They somehow fail to hold any particular topological property (recall that everything in computers is discrete) so being mathematically rigorous with DNN calls for a bad migraine or will yield a very vague theory (see for example K. Weihrauch and computable analysis). 
However, people like Yann LeCun and his energy based approach are onto something (my humble opinion) and there is a whole research field called explainability theory (kind of like psychology of DNN) where math plays an important role in developing a sort of psychiatry for Deep Neural Networks. 
A: Chris Olah has a great blog post on how topology relates to machine learning ("machine learning untangles highly kneaded spaces").
I will let him summarize:

While it is challenging to understand the behavior of deep neural networks in general, it turns out to be much easier to explore low-dimensional deep neural networks – networks that only have a few neurons in each layer. In fact, we can create visualizations to completely understand the behavior and training of such networks. This perspective will allow us to gain deeper intuition about the behavior of neural networks and observe a connection linking neural networks to an area of mathematics called topology.
A number of interesting things follow from this, including fundamental lower-bounds on the complexity of a neural network capable of classifying certain datasets.

His blog also has posts on other specific types of deep neural networks such as "convolutional neural networks", but I haven't read those.
A: Tommy Poggio et al's s MIT course seems great.
A: I have a blog post which discusses some of the connections between deep learning and advanced theoretical physics such as spin funnels and the renormalization group
https://calculatedcontent.com/2015/03/25/why-does-deep-learning-work/
https://calculatedcontent.com/2015/04/01/why-deep-learning-works-ii-the-renormalization-group/
