5
$\begingroup$

I apologize for my ignorance, but hope that someone would provide some pointers to what I am sure is a reasonably well-developed body of theory. Consider $C^\infty(U,V)$ where $U \subset R^k$ and $V \subset R^l$ - I am reasonably sure that this space can be given the structure of a an infinite dimensional manifold. Suppose now that I single out some well-defined class of functions, each element of which is specified by a finite collection of parameters. In particular, I am interested in feed-forward neural networks of fixed architecture, so that the weights and biases give a parametrization, and I would like to think of training as a curve in this class of functions. Does such a collection have the structure of a non-compact finite-dimensional embedded submanifold, in some topology on $C^\infty(U,V)$? What are some references that might cover such topics? I have had standard graduate courses in differential and Riemannian geometry, but am woefully ignorant in functional analysis. I would be grateful for any pointers to sources that might be relevant.

@MattF., I am interested in finding a more natural notion of distance between networks (i.e., functions) than distance between the vectors of parameters in Euclidean space. In fact, I would like to consider the path length (in a space appropriate to the problem - i.e., not just the parameter space) during training as a principled measure of how much the model changes from its initialization (a` la Riemannian geometry). I've read a bit of the information geometry approach, but I'm not sold on the association of a normal probability measure to a neural network (sure, you can do that, but it seems to be choice of convenience more than any sort of rational choice). As the evolution (during training) is a curve in a (presumably) non-flat finite-dimensional submanifold, exploring choices of a Riemannian metric seems like an avenue to consider - maybe one could compute something explicit or find an approximation that could be computed. Still just trying to find a way to think about these things.

$\endgroup$
3
  • 4
    $\begingroup$ I'm not at all an expert, but you might want to take a look at Chapter IX of Kriegl and Michor's The Convenient Setting of Global Analysis? $\endgroup$ Commented Jul 10, 2018 at 20:35
  • 2
    $\begingroup$ The space of parameters is probably R^n — what conceptual gain do you hope for from embedding it into an infinite-dimensional manifold? $\endgroup$
    – user44143
    Commented Jul 11, 2018 at 5:56
  • $\begingroup$ @Matt, I've added a bit more motivation to my post. $\endgroup$ Commented Jul 11, 2018 at 16:01

2 Answers 2

3
$\begingroup$

Lets assume that $U$ is a compact submanifold of $\mathbb R^k$, so that we do not need to worry about boundary conditions and things happening at infinity. Then $C^\infty(U, V)$ is a smooth Fréchet manifold (as explained in the nlab for example).

So, now you are given a finite-dimensional manifold $P$ (which in your example are the weights and biases) and a map $\phi: P \to C^\infty(U, V)$ (which gives you the parametrization of the class of functions you are interested in). By Theorem H in Glöckner's "Fundamentals of submersions and immersions between infinite-dimensional manifolds", the set $\phi(P)$ is a submanifold of $C^\infty(U, V)$ if the differential of $\phi$ is injective at every point of $P$ (this is of course the usual condition for immersions known from finite-dimensional differential geometry; but note that the target manifold is infinite-dimensional so one has to be more careful). If this is the case in your example, depends, of course, on the specific choice of parameterization $\phi$.

By the way, the Riemannian geometry of function spaces is an active topic of research, especially with view towards applications in shape analysis (see e.g. google scholar for related work).

$\endgroup$
1
$\begingroup$

When working with these infinite-dimensional objects, I think it is important to get the rough ideas in place before bringing out the heavy machinery of infinite-dimensional manifolds. Since you mentioned information geometry, it is instructive to study how for example the finite-dimensional (parametric) statistical manifolds with Fisher information metric are embedded in the manifold of smooth densities with the Fisher-Rao metric. The perhaps surprising point is that many formulae can be guessed by rather heuristic calculations. For statistical manifolds many different models have been proposed, and I would not be surprised if the same is true for deterministic mapping spaces.

Regarding your question about machine learning, what you want is perhaps more than a smooth embedding of the parametrized family. I think that in order to get meaningful geometric notions on the submanifold, you need an isometric embedding such that the square of the metric is smooth enough to give you a Riemannian metric (or at least sub-Riemannian) to work with. Of course if you already have a smooth embedding, you naturally obtain the isometric embedding by pulling back some metric from the function space. Coming back to the comment above about getting the rough ideas first: a useful first step is to define some metric on the function space, pull it back, square it, and take its hessian. This works heuristically almost without any special machinery.

Since I have lately been working on these problems, we should maybe get in touch; maybe there is potential for collaboration.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .