Is there any paper which summarizes the mathematical foundation of deep learning?

Now, I am studying about the mathematical background of deep learning. However, unfortunately I cannot know to what extent theory of neural network is mathematically proved. Therefore, I want some paper which review the historical stream of neural network theory based on mathematical foundation, especially in terms of learning algorithms (convergence), and NN’s generalization ability and the NN’s architecture (why deep is good?) If you know, please let me know the name of the paper.

For your reference, let me write down some papers I read.

- Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems, 2(4), 303-314.
Hornik, K., Stinchcombe, M., \& White, H. (1989). Multilayer feedforward networks are universal approximators. Neural networks, 2(5), 359-366.

Funahashi, K. I. (1989). On the approximate realization of continuous mappings by neural networks. Neural networks, 2(3), 183-192.

- Leshno, M., Lin, V. Y., Pinkus, A., \& Schocken, S. (1993). Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 6(6), 861-867.
- Mhaskar, H. N., \& Micchelli, C. A. (1992). Approximation by superposition of sigmoidal and radial basis functions. Advances in Applied mathematics, 13(3), 350-373.
- Delalleau, O., \& Bengio, Y. (2011). Shallow vs. deep sum-product networks. In Advances in Neural Information Processing Systems (pp. 666-674). Telgarsky, M. (2016). Benefits of depth in neural networks. arXiv preprint arXiv:1602.04485.
- Barron, A. R. (1993). Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information theory, 39(3), 930-945.
- Mhaskar, H. N. (1996). Neural networks for optimal approximation of smooth and analytic functions. Neural computation, 8(1), 164-177.
- Lee, H., Ge, R., Ma, T., Risteski, A., \& Arora, S. (2017). On the ability of neural nets to express distributions. arXiv preprint arXiv:1702.07028.
- Bartlett, P. L., \& Maass, W. (2003). Vapnik-Chervonenkis dimension of neural nets. The handbook of brain theory and neural networks, 1188-1192.
- Kawaguchi, K. (2016). Deep learning without poor local minima. In Advances in Neural Information Processing Systems (pp. 586-594).
- Kingma, D. P., \& Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
- Duchi, J., Hazan, E., \& Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul), 2121-2159.
- Tieleman, T., \& Hinton, G. (2012). Lecture 6.5-RMSProp, COURSERA: Neural networks for machine learning. University of Toronto, Technical Report.
- Zeiler, M. D. (2012). ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701.
- Yun, C., Sra, S., \& Jadbabaie, A. (2017). Global optimality conditions for deep neural networks. arXiv preprint arXiv:1707.02444.
- Zeng, J., Lau, T. T. K., Lin, S., \& Yao, Y. (2018). Block Coordinate Descent for Deep Learning: Unified Convergence Guarantees. arXiv preprint arXiv:1803.00225.
- Weinan, E. (2017). A proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1), 1-11. Li, Q., Chen, L., Tai, C., \& Weinan, E. (2017). Maximum principle based algorithms for deep learning. The Journal of Machine Learning Research, 18(1), 5998-6026.
- Zhang, C., Bengio, S., Hardt, M., Recht, B., \& Vinyals, O. (2016). Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530.
- Kawaguchi, K., Kaelbling, L. P., \& Bengio, Y. (2017). Generalization in deep learning. arXiv preprint arXiv:1710.05468.