With this question I'd like to have a recollection of theoretical rigorous results on neural networks. 

I'd like to have results that have been settled, as opposed to hypothesis. As an example, [this][2] paper proposes, under certain assumptions, that any two sets of weights $w_1,w_2$ that achieve a global minimum on the loss are connected by a path along on which the loss is constant and equal to the global minimum; in other words, the global minima are connected. That is a very interesting result, for which it would be nice to have proof, but the paper does not provide one.

An interesting result was [this][1] question I found here about the universal approximation theorem on neural networks in general topological groups other than just $\mathbb{R}^n$, and they do provides proofs. This is the kind of result I am looking for. So, if you know of results like this, please share. 


  [1]: https://mathoverflow.net/questions/404995/neural-networks-over-gadgets-other-than-mathbbr
  [2]: https://proceedings.neurips.cc/paper/2018/file/be3087e74e9100d4bc4c6268cdbe8456-Paper.pdf