Structures of the space of neural networks

Question

A neural network can be considered as a function

$$\mathbf{R}^m\to\mathbf{R}^n\quad \text{by}\quad x\mapsto w_N\sigma(h_{N-1}+w_{N-1}\sigma(\dotso h_2+w_2\sigma(h_1+w_1 x)\dotso)),$$

where the $w_i$ are linear functions (matrices) $\mathbf{R}^{d_{i-1}}\to\mathbf{R}^{d_i}$, the $h_i\in \mathbf{R}^{d_i}$, with $d_0=m$ and $d_N=n$, and $\sigma\colon\mathbf{R}\to\mathbf{R}$ is a non-linear function (e.g. sigmoid), which in the formula above must actually be understood as $\sigma\oplus\dotsb\oplus\sigma$ an appropriate number of times.

There are several studies in the literature (e.g. https://doi.org/10.1016%2F0893-6080%2891%2990009-T) proving that the set of such functions (for fixed $N$ and $\{d_i\}$) is dense in other function spaces, such as measurable-function or continuous-function spaces. Other studies focus on how well functions in this set approximate functions in other sets, according to various measures.

I would like to know, instead, what kind of mathematical structures the set of such functions enjoys, either for fixed or for variable $N$ and $\{d_i\}$. For example, is it a vector space? (answer seems to be yes if the $\{d_i\}$ aren't fixed) Is it a ring under function composition or under some other operation? Is it a convex set? – And similar questions.

I'm thankful to anyone who can provide some literature about such questions.

Update – example: this brilliant study by Petersen, Raslan, Voigtlaender https://arxiv.org/abs/1806.08459 answers the question about convexity.

Answer 1

In information geometry, people study structures of Riemannian manifolds with dual affine connections on sets of neural networks. (The metric measures how close neural networks are in their input-output behaviour.) Riemannian geometry can then be used to study learning algorithms (like gradient descent methods). Here are a couple of references, more can be found by searching for the tags "information geometry" and "neural networks".

• S. Amari. Information geometry of the EM and em algorithms for neural networks. Neural Networks 8 (1995), 1379-1408.

• S. Amari, K. Kurata and H. Nagaoka. Information geometry of Boltzmann machines. IEEE Transactions on Neural Networks, vol. 3, no. 2, (1992), 260-271.

• S. Amari. Information Geometry of Neural Networks — An Overview —. In: Ellacott S.W., Mason J.C., Anderson I.J. (eds) Mathematics of Neural Networks. Operations Research/Computer Science Interfaces Series, vol 8. Springer, Boston, MA, 1997. (link to Springer website, paywall)

I don't know about recent developments, all the references above are well before the deep learning revolution.

Answer 2

I would like to argue that the space of neural networks is a category with finite products, or more concretely a Lawvere theory. This expresses an important piece of structure, namely how neural networks can be composed to form larger networks and be decomposed into individual neurons. In order to talk about other things such as approximation results, one probably wants to equip this category with some additional structure, such as metrics on the hom-sets.

The collection of all smooth functions between Euclidean spaces $\mathbb{R}^n \to \mathbb{R}^m$ forms a category whose objects are the natural numbers $n\in\mathbb{N}$. Due to the existence of finite products, this category is symmetric monoidal: we can compose morphisms not only sequentially, but also in parallel. The nLab denotes this category as $\mathsf{CartSp_{smooth}}$.

Now let us restrict to those functions $\mathbb{R}^n\to\mathbb{R}^m$ which can be implemented by neural networks. This class of functions is closed both under sequential and under parallel composition; it therefore forms a symmetric monoidal subcategory $\mathsf{CartSp_{neural}}\subset\mathsf{CartSp_{smooth}}$. In fact, by the very definition of a neural network, this is the symmetric monoidal subcategory generated by the sigmoid function together with all linear maps. Since it contains the product projections $\mathbb{R}^{m+n}\to\mathbb{R}^m$ and $\mathbb{R}^{m+n}\to\mathbb{R}^n$ in particular, it is easy to see that it is actually a sub-Lawvere theory of $\mathsf{CartSp_{smooth}}$. It could be interesting to work out what its models are, and how they relate to smooth algebras!

For closely related developments, see Fong and Spivak's paper Backprop as Functor.