29
$\begingroup$

Is there any paper which summarizes the mathematical foundation of deep learning?

Now, I am studying about the mathematical background of deep learning. However, unfortunately I cannot know to what extent theory of neural network is mathematically proved. Therefore, I want some paper which review the historical stream of neural network theory based on mathematical foundation, especially in terms of learning algorithms (convergence), and NN’s generalization ability and the NN’s architecture (why deep is good?) If you know, please let me know the name of the paper.

For your reference, let me write down some papers I read.

  • Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4), 303-314.

  • Hornik, K., Stinchcombe, M., \& White, H. (1989). Multilayer feedforward networks are universal approximators. Neural networks, 2(5), 359-366.

  • Funahashi, K. I. (1989). On the approximate realization of continuous mappings by neural networks. Neural networks, 2(3), 183-192.

  • Leshno, M., Lin, V. Y., Pinkus, A., \& Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6(6), 861-867.
  • Mhaskar, H. N., \& Micchelli, C. A. (1992). Approximation by superposition of sigmoidal and radial basis functions. Advances in Applied mathematics, 13(3), 350-373.
  • Delalleau, O., \& Bengio, Y. (2011). Shallow vs. deep sum-product networks. In Advances in Neural Information Processing Systems (pp. 666-674). Telgarsky, M. (2016). Benefits of depth in neural networks. arXiv preprint arXiv:1602.04485.
  • Barron, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3), 930-945.
  • Mhaskar, H. N. (1996). Neural networks for optimal approximation of smooth and analytic functions. Neural computation, 8(1), 164-177.
  • Lee, H., Ge, R., Ma, T., Risteski, A., \& Arora, S. (2017). On the ability of neural nets to express distributions. arXiv preprint arXiv:1702.07028.
  • Bartlett, P. L., \& Maass, W. (2003). Vapnik-Chervonenkis dimension of neural nets. The handbook of brain theory and neural networks, 1188-1192.
  • Kawaguchi, K. (2016). Deep learning without poor local minima. In Advances in Neural Information Processing Systems (pp. 586-594).
  • Kingma, D. P., \& Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  • Duchi, J., Hazan, E., \& Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul), 2121-2159.
  • Tieleman, T., \& Hinton, G. (2012). Lecture 6.5-RMSProp, COURSERA: Neural networks for machine learning. University of Toronto, Technical Report.
  • Zeiler, M. D. (2012). ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701.
  • Yun, C., Sra, S., \& Jadbabaie, A. (2017). Global optimality conditions for deep neural networks. arXiv preprint arXiv:1707.02444.
  • Zeng, J., Lau, T. T. K., Lin, S., \& Yao, Y. (2018). Block Coordinate Descent for Deep Learning: Unified Convergence Guarantees. arXiv preprint arXiv:1803.00225.
  • Weinan, E. (2017). A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1), 1-11. Li, Q., Chen, L., Tai, C., \& Weinan, E. (2017). Maximum principle based algorithms for deep learning. The Journal of Machine Learning Research, 18(1), 5998-6026.
  • Zhang, C., Bengio, S., Hardt, M., Recht, B., \& Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530.
  • Kawaguchi, K., Kaelbling, L. P., \& Bengio, Y. (2017). Generalization in deep learning. arXiv preprint arXiv:1710.05468.
Improve this question
$\endgroup$
2
  • 17
    $\begingroup$ In my opinion deep learning does not have a proper mathematical foundation yet - there are no convergence theorems which explain the observed superiority of these models to previous techniques, there is little theory to organize the strengths and weaknesses of the various different architectures, and there are known examples of toy machine learning problems for which the leading models fail spectacularly for unknown reasons. Compared with other branches of statistics, it's kind of a disaster. $\endgroup$
    – Paul Siegel
    Commented Oct 11, 2018 at 13:05
  • $\begingroup$ Something those papers don't seem to address : usually there is not only the model which is fed with the ideal training data, but also a model for designing the real-life training data. $\endgroup$
    – reuns
    Commented Oct 11, 2018 at 21:01

1 Answer 1

Reset to default
13
$\begingroup$

Mathematics of Deep Learning (2017)

This tutorial will review recent work that aims to provide a mathematical justification for several properties of deep networks, such as global optimality, geometric stability, and invariance of the learned representations.

Deep Learning: An Introduction for Applied Mathematicians (2018)

This article provides a very brief introduction to the basic ideas that underlie deep learning from an applied mathematics perspective. Our target audience includes postgraduate and final year undergraduate students in mathematics who are keen to learn about the area.

Improve this answer
$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .