Relation between information geometry and geometric deep learning

Question

Disclaimer: This is a cross-post from a very similar question on math.SE. I allowed myself to post it here after reading this meta post about cross-posting between mathoverflow and math.SE, I did try both the 1. and 2. suggestions of the accepted answer (suggestion 2. - asking for help on a mathoverflow meta dedicated post - was impossible because I lacked the reputation necessary to answer on mathoverflow meta).

I'm currently working on information geometry (IG) and geometric deep learning (GDL). As I started without specific knowledge of both, their respective names led me to believe for a short and naive period that GDL was defined by the use of IG notions in deep learning. This now appears to me as substantially inaccurate, but since there are indeed various connections, I would like to clarify the relation between them.

In Geometric deep learning: going beyond Euclidean data, GDL is defined as:

Geometric deep learning is an umbrella term for emerging techniques attempting to generalize (structured) deep neural models to non-Euclidean domains such as graphs and manifolds.

The article later defines Riemannian manifolds and metrics, calculus on manifolds, etc to complete the toolkit needed to build a machine learning favorable environment.

On the other hand, in Amari's Information Geometry and Its Applications, IG is described in the following paragraph:

Information geometry has emerged from studies of invariant geometrical structure involved in statistical inference. It defines a Riemannian metric together with dually coupled affine connections in a manifold of probability distributions. These structures play important roles not only in statistical inference but also in wider areas of information sciences, such as machine learning, signal processing, optimization, and even neuroscience, not to mention mathematics and physics.

In the same book, Amari mentions a neural manifold, containing all neural networks:

The set of all such networks forms a manifold, where matrix $W =w_{ji}$ is a coordinate system.

The underlying question is: to what extent is GDL related to IG ?

If I'm not mistaken, working on a riemannian manifold with a neural network, and relying on a riemannian gradient for its training, implies one is doing geometric deep learning, but not necessarily information geometry. An example of such GDL/non-IG neural network is SPD Net, which relies on SPD matrices for intermediate representations, defines new transformations aiming at keeping representations on a manifold (hence a bilinear mapping layer $W.X.W^T$ with $W$ belonging to a compact Stiefel manifold - cf. BiMap Layer section of the previous article), and depends on a riemannian gradient. It seems similar to IG since we speak of riemannian manifold and riemannian gradient, but it doesn't match the manifold of probability distributions aspect. A similar SPD matrices neural network architecture can be found in this paper.

My current understanding is that GDL and IG are closely related since they happen to rely on similar mathematical objects, and perhaps more importantly, that in both cases we try to reach the right representation of a "latent subspace". The learning mechanisms of GDL then owe their success to optimization on the well chosen manifolds using gradients possibly defined in IG, the manifolds ideally describing a subspace where all possible data is living, not only the samples available for training. However, this relation is limited to specific cases of GDL, where a neural network uses manifolds and metrics also used in IG. One can notice that this completely excludes, among others, the graph-based part of GDL (?).

Finally, we could still find a direct relation between GDL and IG if we used neural networks to work on manifolds of probability distributions, which are the very objects that IG targets. That seems rather relevant for generative models, but it's a very specific case of GDL for which the high usefulness of IG is contextual.

Related AI SE post: What is geometric deep learning?, my answer there could be inaccurate depending on what the community will answer here.

Mathoverflow addition: this mathoverflow question looking for IG references made me discover Amari's paper "Information geometry in optimization, machine learning and statistical inference" which is closely related to my question, but predates the definition of GDL in Geometric deep learning: going beyond Euclidean data, and thus ignores its position with respect to the latter.

It doesn't seem that inappropriate to post this "boundaries definition" question on this SE Q&A since GDL is a recent and active research field (I've seen a SPD neural networks paper accepted at NIPS 2019 for example). If, after all, it still is out of place, sincere apologies, I'll just wait for answers on math.SE.

EDIT: related paper establishing a link between Deep Learning (not specific to GDL) and IG through the Fisher-Rao norm: "Fisher-Rao Metric, Geometry, and Complexity of Neural Networks".

Answer 1

The fields you're talking about are typically concerned with two different geometric spaces:

• The space of input data to a neural network (geometric deep learning)
• The parameter space of all neural networks with a given architecture (information geometry)

Many natural applications of neural networks involve input data with a discrete Euclidean-type structure: 1D for time series, 2D for images or audio, 3D for video. That "Geometric Deep Learning" paper discusses applying neural networks to input data with other types of geometry, such as graphs and networks. A central problem is figuring out the right architecture to handle a particular type of data.

On the other hand, suppose you're studying the question of training a particular neural network. That is, you have a specific architecture in mind, let's say with $n$ weight (and maybe bias) parameters, where any given set of parameters may be viewed as a point in $\mathbb{R}^n$. When you study the dynamics of training, it can be useful to think about different metrics on this space. For example, some common regularization methods rely on $L_1$ or $L_2$ norms. The "information geometry" line of work looks at other metrics, with the goal of capturing more sophisticated concepts of network capacity, invariances to certain transformations, etc. A paper with a relatively brief, self-contained exposition is Fisher-Rao Metric, Geometry, and Complexity of Neural Networks.

To sum up: Geometric deep learning is concerned with problems where the domain, or input data, is far from being modeled by a standard Euclidean space. Information geometry is traditionally used to analyze dynamics on a neural network parameter space ($\mathbb{R}^n$, but with a non-Euclidean metric). So in that sense, they are conceptually distinct. However, they both use similar mathematics, and certainly both could arise in studying a particular neural network.