Theoretical results on neural networks

Question

With this question I'd like to have a recollection of theoretical rigorous results on neural networks.

I'd like to have results that have been settled, as opposed to hypothesis. As an example, this paper proposes, under certain assumptions, that any two sets of weights $w_1,w_2$ that achieve a global minimum on the loss are connected by a path along on which the loss is constant and equal to the global minimum; in other words, the global minima are connected. That is a very interesting result, for which it would be nice to have proof, but the paper does not provide one.

An interesting result was this question I found here about the universal approximation theorem on neural networks in general topological groups other than just $\mathbb{R}^n$, and they do provides proofs. This is the kind of result I am looking for. So, if you know of results like this, please share.

Edit 1: As some have pointed out, the question is indeed broad. That is kind of goal of the question though. I'd like for people to share papers/results that I (and others) are not aware. The only requisite is that it is well-settled result, and not a hypothesis.

Answer 1

Your question is a bit too broad, but here is something you may want to read, if you are interested in the Mathematical Analysis of Deep Learning: The Modern Mathematics of Deep Learning, by Berner et al., arXiv:2105.04026.

Answer 2

ICLR 2021 has contributions that could qualify as "rigorous results", one you may like is Minimum Width for Universal Approximation.

The universal approximation property of width-bounded networks is characterized in terms of the input dimension $d_x$ and the output dimension $d_y$. The minimum width required for the universal approximation of $L^p$ functions is exactly $\max \{d_x+1,d_y\}.$

Answer 3

The Representer Theorem by Michael Unser has recently unveiled explicit connections between deep NNs (using ReLUs as nonlinearities I believe) and splines.

One core idea is that both the linear operators and ReLUs can be seen as piecewise linear functions (linear splines), so all forward and backward operations, and the resulting representations are closed in that set.

This allows to connect DNNs with L1 methods, and therefore with some of the advances in Compressed Sensing from the last decades.

Not only that, it also leads to novel architectures with promising properties, like B-spline networks:

We develop an efficient computational solution to train deep neural networks (DNN) with free-form activation functions. To make the problem well-posed, we augment the cost functional of the DNN by adding an appropriate shape regularization: the sum of the second-order total variations of the trainable nonlinearities. The representer theorem for DNNs tells us that the optimal activation functions are adaptive piecewise-linear splines, which allows us to recast the problem as a parametric optimization. The challenging point is that the corresponding basis functions (ReLUs) are poorly conditioned and that the determination of their number and positioning is also part of the problem. We circumvent the difficulty by using an equivalent B-spline basis to encode the activation functions and by expressing the regularization as an l1-penalty. This results in the specification of parametric activation function modules that can be implemented and optimized efficiently on standard development platforms. We present experimental results that demonstrate the benefit of our approach.