A neural network can be considered as a function

$$\mathbf{R}^m\to\mathbf{R}^n\quad \text{by}\quad x\mapsto w_N\sigma(h_{N-1}+w_{N-1}\sigma(\dotso h_2+w_2\sigma(h_1+w_1 x)\dotso)),$$

where the $w_i$ are linear functions (matrices) $\mathbf{R}^{d_{i-1}}\to\mathbf{R}^{d_i}$, the $h_i\in \mathbf{R}^{d_i}$, with $d_0=m$ and $d_N=n$, and $\sigma\colon\mathbf{R}\to\mathbf{R}$ is a non-linear function (e.g. sigmoid), which in the formula above must actually be understood as $\sigma\oplus\dotsb\oplus\sigma$ an appropriate number of times.

There are several studies in the literature (e.g. https://doi.org/10.1016%2F0893-6080%2891%2990009-T) proving that the set of such functions (for fixed $N$ and $\{d_i\}$) is dense in other function spaces, such as measurable-function or continuous-function spaces. Other studies focus on how well functions in this set approximate functions in other sets, according to various measures.

I would like to know, instead, what kind of mathematical structures the set of such functions enjoys, either for fixed or for variable $N$ and $\{d_i\}$. For example, is it a vector space? (answer seems to be yes if the $\{d_i\}$ aren't fixed) Is it a ring under function composition or under some other operation? Is it a convex set? – And similar questions.

I'm thankful to anyone who can provide some literature about such questions.

Update – example: this brilliant study by Petersen, Raslan, Voigtlaender https://arxiv.org/abs/1806.08459 answers the question about convexity.