71
$\begingroup$

I am interested in finding out the math ideas behind the technologies that are under the umbrella of "Deep Learning" or "Deep neural nets".

Most of the papers/books that are often quoted in papers/online as references are not written in a very math-friendly manner. I am specifically referring to the fact that this field is highly interdisciplinary, and the language used (e.g. 'levels', 'stacking networks') are not standard mathematical terminology, but rather very specialized terms.

So I am writing this post to find out if there exists a book or review article written for pure mathematicians about the core mathematical ideas of the whole deep-learning thing.

My hope is that is there is a reference that follows (sort of ) the theorem-lemma-proof format or at least tries to where ever possible, or at least gives some rigorous definitions so that I can make sense of the terminology.

Thank you.

$\endgroup$
  • 7
    $\begingroup$ This is more for physicists but nevertheless looks like it should be pretty insightful: arxiv.org/abs/1410.3831 $\endgroup$ – Steve Huntsman Apr 9 '15 at 14:36
  • 4
    $\begingroup$ Do you mean to specify deep learning (the training of neural networks with multiple hidden layers, primarily developed in the last 9 years) as opposed to more standard techniques in machine learning including things like SVMs, universal representability, and the Vapnik–Chervonenkis dimension? $\endgroup$ – Douglas Zare Apr 9 '15 at 17:10
  • 1
    $\begingroup$ I guess a bit of both, but my main motivation has been to see what the main idea of the new stuff is. I guess I would appreciate good references for both (new and old) from mathematical point of view. $\endgroup$ – aa12 Apr 9 '15 at 20:19
  • 4
    $\begingroup$ Also, one of the challenges to a formal understanding of deep learning methods is that the performance of a method depends both on the model ("architecture") and the parameter estimation algorithm ("training", which is often heuristic). Many engineering-oriented treatments conflate these two elements in ways that can be confusing. $\endgroup$ – R Hahn Apr 10 '15 at 3:38
  • 3
    $\begingroup$ Have a look at this book: iro.umontreal.ca/~bengioy/dlbook --- that book is not so mathematical, but at least will show several kinds of the math that shows up in "learning" (probability, statistics, functional analysis, graph theory, optimization, etc.) $\endgroup$ – Suvrit Apr 12 '15 at 15:56

11 Answers 11

22
$\begingroup$

If you have time, I highly recommend this Coursera course.

The videos are available for free and are truly excellent. The teacher is Geoffrey Hinton, who is one of the main players in the area, and he does an excellent job of providing both clear definitions and useful intuition.

In general, I wouldn't expect to see perfect theorem-lemma-proof exposition of deep learning anywhere, simply because the math hasn't caught up to real-world practice. More typical is a clean analysis of an idealized system, which is then related to a real system by a heuristic argument. In other words, this is an area that could use attention from mathematicians!

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The link is not functioning. $\endgroup$ – Idonknow Apr 1 at 4:59
11
$\begingroup$

Chris Olah has a great blog post on how topology relates to machine learning ("machine learning untangles highly kneaded spaces").

I will let him summarize:

While it is challenging to understand the behavior of deep neural networks in general, it turns out to be much easier to explore low-dimensional deep neural networks – networks that only have a few neurons in each layer. In fact, we can create visualizations to completely understand the behavior and training of such networks. This perspective will allow us to gain deeper intuition about the behavior of neural networks and observe a connection linking neural networks to an area of mathematics called topology.

A number of interesting things follow from this, including fundamental lower-bounds on the complexity of a neural network capable of classifying certain datasets.

His blog also has posts on other specific types of deep neural networks such as "convolutional neural networks", but I haven't read those.

$\endgroup$
10
$\begingroup$

Tommy Poggio et al's s MIT course seems great.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ The poster asked about math specific to deep learning, not to general machine learning / statistical learning theory. So this does not really answer the question I think. $\endgroup$ – Suvrit Apr 9 '15 at 22:19
  • $\begingroup$ @Suvrit He asked for a reasonably jargon free rigorous explanation of the field, and this is it, I think. $\endgroup$ – Igor Rivin Apr 9 '15 at 22:49
  • $\begingroup$ Yes, but a description of "Deep Learning" not of the field of machine learning --- but given the OPs comments, it seems that he/she may not be sure what they want. $\endgroup$ – Suvrit Apr 10 '15 at 2:30
  • 2
    $\begingroup$ @Suvrit: The last two lectures are specifically on deep learning, and they provide a lot more connections to mathematics than most treatments of deep learning. $\endgroup$ – Douglas Zare Apr 12 '15 at 15:13
  • 1
    $\begingroup$ @DouglasZare: Maybe this book may appeal a bit more to the OP: iro.umontreal.ca/~bengioy/dlbook $\endgroup$ – Suvrit Apr 12 '15 at 15:54
10
$\begingroup$

I have a blog post which discusses some of the connections between deep learning and advanced theoretical physics such as spin funnels and the renormalization group

http://charlesmartin14.wordpress.com/2015/03/25/why-does-deep-learning-work/

http://charlesmartin14.wordpress.com/2015/04/01/why-deep-learning-works-ii-the-renormalization-group/

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

Please have a look at Jason Morton's work on Mathematics of Deep Learning. It is quite mathematically rigorous if that's what is you are asking for.

Link: http://www.jasonmorton.com/publications.html

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

Recently uploaded paper in Arxiv (1512.06293). This paper formalize concepts and proves them. And few others that you might check out:

  1. J. Bruna and S. Mallat, “Invariant Scattering Convolution Networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1872–1886, Aug. 2013.

    • for Covnet
  2. A. Choromanska, M. Henaff, M. Mathieu, G. B. Arous, and Y. LeCun, “The Loss Surfaces of Multilayer Networks,” arXiv:1412.0233 [cs], Nov. 2014.

    • Spin Glasses Theory
    • for Deep MLP and Convnet
  3. P. Mehta and D. J. Schwab, “An exact mapping between the Variational Renormalization Group and Deep Learning,” arXiv:1410.3831 [cond-mat, stat], Oct. 2014.

    • for Restricted Boltzmann Machine, Deep Belief Network and Deep Boltzmann Machine.
$\endgroup$
3
$\begingroup$

I just found this paper-https://arxiv.org/pdf/1801.05894.pdf, which introduces deep learning in a mathematically sound manner, specially for computations of backpropagation etc. As a mathematician who worked in machine learning but not deep learning, I've noticed that the tools that're often needed for machine learning are linear algebra (with a bit of functional analysis), probability and optimization.

Apart from the above, there're also some literature using differential geometry and triangle meshes, e.g. https://arxiv.org/pdf/1611.08097.pdf, although I'm not quite sure whether they're being used in industrial applications. My suspicion comes from the fact that in general most of machine learning that uses differential geometry, e.g. manifold learning, is mostly useless outside academia, hence in real world, as real life observations don't satisfy smooth manifold assumption. However, I'd be glad, as a mathematician, to be proven wrong, in the sense if you could point out some true industrial use of geometric machine learning/geometric deep learning to solve realistic problems.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

I worked with neural nets in the 80's and haven't kept up with the literatue, so I don't have any links. However, I came away with the conclusion that training neural nets is essentially training a digital computer to behave like an analog computer. So, examining the mathematics pertaining to analog circuits might give some insight in this area.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I find this interesting. Is it meant to be kind of a handwaving analogy or an ansatz that can be made rigorous? $\endgroup$ – Delio Mugnolo Feb 14 '16 at 21:09
  • $\begingroup$ @DelioMugnolo, When I experimented with neural nets (as a hobby), I saw the analogy to bridge networks---plug in an unknown resistor (say) and read its value on a meter. The pieces of the bridge would be somewhat like the nodes of a neural net. It is only my take on the process. $\endgroup$ – Fred Kline Feb 15 '16 at 7:23
  • $\begingroup$ @DelioMugnolo, additional observation---D-WAVE's quantum computer is analog. $\endgroup$ – Fred Kline Feb 15 '16 at 7:35
  • $\begingroup$ Thanks. Do D-Wave computers have anything to do with neural nets? $\endgroup$ – Delio Mugnolo Feb 15 '16 at 8:06
1
$\begingroup$

There's an ongoing course taught by Elchanan Mossel at MIT that you might find helpful. It really focuses on the things we can actually prove about deep learning, which may be mathematically appealing to you. The homepage is here.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I'd say that deep learning (from a mathematician's perspective) is a HOT MESS. People are jumping through hoops trying to increase accuracy (sometimes just by decimals) introducing all sorts of stuff to models. They are like mad scientists, messing those poor neural networks.

Though the architecture of deep neural networks is deceptively simple, their structural composition just cannot explain everything. They depend on gazillion parameters, and a gazillion functions. Their dynamics just escapes traditional chaos theory. Bringing to the table stuff like functional analysis just doesn't cut it. Datasets are huge and unstructured. They somehow fail to hold any particular topological property (recall that everything in computers is discrete) so being mathematically rigorous with DNN calls for a bad migraine or will yield a very vague theory (see for example K. Weihrauch and computable analysis).

However, people like Yann LeCun and his energy based approach are onto something (my humble opinion) and there is a whole research field called explainability theory (kind of like psychology of DNN) where math plays an important role in developing a sort of psychiatry for Deep Neural Networks.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Eldad Haber at the University of British Columbia is exploring connections between neural networks, and dynamical systems:

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.