4
$\begingroup$

I apologize for my ignorance, but hope that someone would provide some pointers to what I am sure is a reasonably well-developed body of theory. Consider $C^\infty(U,V)$ where $U \subset R^k$ and $V \subset R^l$ - I am reasonably sure that this space can be given the structure of a an infinite dimensional manifold. Suppose now that I single out some well-defined class of functions, each element of which is specified by a finite collection of parameters. In particular, I am interested in feed-forward neural networks of fixed architecture, so that the weights and biases give a parametrization, and I would like to think of training as a curve in this class of functions. Does such a collection have the structure of a non-compact finite-dimensional embedded submanifold, in some topology on $C^\infty(U,V)$? What are some references that might cover such topics? I have had standard graduate courses in differential and Riemannian geometry, but am woefully ignorant in functional analysis. I would be grateful for any pointers to sources that might be relevant.

@MattF., I am interested in finding a more natural notion of distance between networks (i.e., functions) than distance between the vectors of parameters in Euclidean space. In fact, I would like to consider the path length (in a space appropriate to the problem - i.e., not just the parameter space) during training as a principled measure of how much the model changes from its initialization (a` la Riemannian geometry). I've read a bit of the information geometry approach, but I'm not sold on the association of a normal probability measure to a neural network (sure, you can do that, but it seems to be choice of convenience more than any sort of rational choice). As the evolution (during training) is a curve in a (presumably) non-flat finite-dimensional submanifold, exploring choices of a Riemannian metric seems like an avenue to consider - maybe one could compute something explicit or find an approximation that could be computed. Still just trying to find a way to think about these things.

$\endgroup$
  • 4
    $\begingroup$ I'm not at all an expert, but you might want to take a look at Chapter IX of Kriegl and Michor's The Convenient Setting of Global Analysis? $\endgroup$ – Branimir Ćaćić Jul 10 '18 at 20:35
  • 2
    $\begingroup$ The space of parameters is probably R^n — what conceptual gain do you hope for from embedding it into an infinite-dimensional manifold? $\endgroup$ – Matt F. Jul 11 '18 at 5:56
  • $\begingroup$ @Matt, I've added a bit more motivation to my post. $\endgroup$ – Dave Johannsen Jul 11 '18 at 16:01
3
$\begingroup$

Lets assume that $U$ is a compact submanifold of $\mathbb R^k$, so that we do not need to worry about boundary conditions and things happening at infinity. Then $C^\infty(U, V)$ is a smooth Fréchet manifold (as explained in the nlab for example).

So, now you are given a finite-dimensional manifold $P$ (which in your example are the weights and biases) and a map $\phi: P \to C^\infty(U, V)$ (which gives you the parametrization of the class of functions you are interested in). By Theorem H in Glöckner's "Fundamentals of submersions and immersions between infinite-dimensional manifolds", the set $\phi(P)$ is a submanifold of $C^\infty(U, V)$ if the differential of $\phi$ is injective at every point of $P$ (this is of course the usual condition for immersions known from finite-dimensional differential geometry; but note that the target manifold is infinite-dimensional so one has to be more careful). If this is the case in your example, depends, of course, on the specific choice of parameterization $\phi$.

By the way, the Riemannian geometry of function spaces is an active topic of research, especially with view towards applications in shape analysis (see e.g. google scholar for related work).

$\endgroup$
1
$\begingroup$

When working with these infinite-dimensional objects, I think it is important to get the rough ideas in place before bringing out the heavy machinery of infinite-dimensional manifolds. Since you mentioned information geometry, it is instructive to study how for example the finite-dimensional (parametric) statistical manifolds with Fisher information metric are embedded in the manifold of smooth densities with the Fisher-Rao metric. The perhaps surprising point is that many formulae can be guessed by rather heuristic calculations. For statistical manifolds many different models have been proposed, and I would not be surprised if the same is true for deterministic mapping spaces.

Regarding your question about machine learning, what you want is perhaps more than a smooth embedding of the parametrized family. I think that in order to get meaningful geometric notions on the submanifold, you need an isometric embedding such that the square of the metric is smooth enough to give you a Riemannian metric (or at least sub-Riemannian) to work with. Of course if you already have a smooth embedding, you naturally obtain the isometric embedding by pulling back some metric from the function space. Coming back to the comment above about getting the rough ideas first: a useful first step is to define some metric on the function space, pull it back, square it, and take its hessian. This works heuristically almost without any special machinery.

Since I have lately been working on these problems, we should maybe get in touch; maybe there is potential for collaboration.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.