# Theoretical results on neural networks

With this question I'd like to have a recollection of theoretical rigorous results on neural networks.

I'd like to have results that have been settled, as opposed to hypothesis. As an example, this paper proposes, under certain assumptions, that any two sets of weights $$w_1,w_2$$ that achieve a global minimum on the loss are connected by a path along on which the loss is constant and equal to the global minimum; in other words, the global minima are connected. That is a very interesting result, for which it would be nice to have proof, but the paper does not provide one.

An interesting result was this question I found here about the universal approximation theorem on neural networks in general topological groups other than just $$\mathbb{R}^n$$, and they do provides proofs. This is the kind of result I am looking for. So, if you know of results like this, please share.

Edit 1: As some have pointed out, the question is indeed broad. That is kind of goal of the question though. I'd like for people to share papers/results that I (and others) are not aware. The only requisite is that it is well-settled result, and not a hypothesis.

• Too broad. Every ML conference of past 10 years has tens of such results. Oct 3 at 19:27
• I am not sure about tens. Many results are hypothesis or are not settled yet. One example is generalization, there have been many papers on that, but no one has really said the final word. Oct 3 at 20:39
• See Sanjeev Arora's page on machine learning with provable guarantees. Oct 4 at 1:06
• I'd also say that the question is much too broad to be useful (a fair comparison maybe "Theoretic results on numerics of PDEs"). You'll end up with a large, but still biased, sample of the possible answers.
– Dirk
Oct 4 at 5:36
• That said, you could focus your question on specific aspects, like expressivity, trainability, or generalization and/or specific architectures (Fully connected, convolutional, ResNet, LSTMs,...) and/or specific tasks (classification, generation, unsupervised learning,...).
– Dirk
Oct 4 at 5:39

Your question is a bit too broad, but here is something you may want to read, if you are interested in the Mathematical Analysis of Deep Learning: The Modern Mathematics of Deep Learning, by Berner et al., arXiv:2105.04026.

ICLR 2021 has contributions that could qualify as "rigorous results", one you may like is Minimum Width for Universal Approximation.

The universal approximation property of width-bounded networks is characterized in terms of the input dimension $$d_x$$ and the output dimension $$d_y$$. The minimum width required for the universal approximation of $$L^p$$ functions is exactly $$\max \{d_x+1,d_y\}.$$

The Representer Theorem by Michael Unser has recently unveiled explicit connections between deep NNs (using ReLUs as nonlinearities I believe) and splines.

One core idea is that both the linear operators and ReLUs can be seen as piecewise linear functions (linear splines), so all forward and backward operations, and the resulting representations are closed in that set.

This allows to connect DNNs with L1 methods, and therefore with some of the advances in Compressed Sensing from the last decades.

Not only that, it also leads to novel architectures with promising properties, like B-spline networks:

We develop an efficient computational solution to train deep neural networks (DNN) with free-form activation functions. To make the problem well-posed, we augment the cost functional of the DNN by adding an appropriate shape regularization: the sum of the second-order total variations of the trainable nonlinearities. The representer theorem for DNNs tells us that the optimal activation functions are adaptive piecewise-linear splines, which allows us to recast the problem as a parametric optimization. The challenging point is that the corresponding basis functions (ReLUs) are poorly conditioned and that the determination of their number and positioning is also part of the problem. We circumvent the difficulty by using an equivalent B-spline basis to encode the activation functions and by expressing the regularization as an l1-penalty. This results in the specification of parametric activation function modules that can be implemented and optimized efficiently on standard development platforms. We present experimental results that demonstrate the benefit of our approach.