I apologize for my ignorance, but hope that someone would provide some pointers to what I am sure is a reasonably well-developed body of theory. Consider $C^\infty(U,V)$ where $U \subset R^k$ and $V \subset R^l$ - I am reasonably sure that this space can be given the structure of a an infinite dimensional manifold. Suppose now that I single out some well-defined class of functions, each element of which is specified by a finite collection of parameters. In particular, I am interested in feed-forward neural networks of fixed architecture, so that the weights and biases give a parametrization, and I would like to think of training as a curve in this class of functions. Does such a collection have the structure of a non-compact finite-dimensional embedded submanifold, in some topology on $C^\infty(U,V)$? What are some references that might cover such topics? I have had standard graduate courses in differential and Riemannian geometry, but am woefully ignorant in functional analysis. I would be grateful for any pointers to sources that might be relevant.

@MattF., I am interested in finding a more natural notion of distance between networks (i.e., functions) than distance between the vectors of parameters in Euclidean space. In fact, I would like to consider the path length (in a space appropriate to the problem - i.e., not just the parameter space) during training as a principled measure of how much the model changes from its initialization (a` la Riemannian geometry). I've read a bit of the information geometry approach, but I'm not sold on the association of a normal probability measure to a neural network (sure, you can do that, but it seems to be choice of convenience more than any sort of rational choice). As the evolution (during training) is a curve in a (presumably) non-flat finite-dimensional submanifold, exploring choices of a Riemannian metric seems like an avenue to consider - maybe one could compute something explicit or find an approximation that could be computed. Still just trying to find a way to think about these things.