How useful is differential geometry and topology to deep learning?

Question

After seeing this article https://www.quantamagazine.org/an-idea-from-physics-helps-ai-see-in-higher-dimensions-20200109/ I wanted to ask myself how useful of an endeavor would it be if one goes through the process of learning differential geometry to deep learning? Perhaps there are other tools as well?

Can concepts such as homotopy be 'appropriately' encodable in deep learning? I feel notions of homotopy invariance if properly defined could be much more valuable than the simple attempt in quantamagazine article. A basic attempt (away from deep learning framework) is in https://arxiv.org/abs/1408.2584.

Answer 1

A "roadmap type" introduction is given by Roger Grosse in Differential geometry for machine learning.

Differential geometry is all about constructing things which are independent of the representation. You treat the space of objects (e.g. distributions) as a manifold, and describe your algorithm in terms of things that are intrinsic to the manifold itself. While you ultimately need to use some coordinate system to do the actual computations, the higher-level abstractions make it easier to check that the objects you're working with are intrinsically meaningful. This roadmap is intended to highlight some examples of models and algorithms from machine learning which can be interpreted in terms of differential geometry. Most of the content in this roadmap belongs to information geometry, the study of manifolds of probability distributions. The best reference on this topic is probably Amari and Nagaoka's Methods of Information Geometry.

Answer 2

You can read this paper:

https://arxiv.org/abs/1805.10451

Although what they say is by no means new - Coifman had had this point of view for the last thirty years at least.

Answer 3

Topological data analysis is already pretty popular in "data science", so much so that there are companies built on this idea. See: 1). https://en.wikipedia.org/wiki/Topological_data_analysis 2).https://web.stanford.edu/class/archive/ee/ee392n/ee392n.1146/lecture/may13/EE392n_TDA_online.pdf 3). https://towardsdatascience.com/a-concrete-application-of-topological-data-analysis-86b89aa27586

Answer 4

Taco Cohen has written some papers that use Differential Geometry, Topology, Guage theory, etc. in Machine learning.

Answer 5

Colah gives a very interesting perspective about deep learning and neural networks in the context of topology. He discusses the "Manifold Hypothesis" which, in short, tries to explain why deep learning is so effective. To read more about the Manifold Hypothesis, Goodfellow has a chapter on it.

On another note, there are interesting articles being published in the context of topological analysis of some "image spaces". For example, Carlsson et al.:

In this study we concentrate on qualitative topological analysis of the local behavior of the space of natural images. To this end, we use a space of 3 by 3 high-contrast patches ℳ. We develop a theoretical model for the high-density 2-dimensional submanifold of ℳ showing that it has the topology of the Klein bottle.

I think learning basic topological ideas and especially all the work that has already been done on the connection of data analysis and topology can be very fruitful (Mind that I am no expert though).

Answer 6

Heat kernels are based on ideas from differential geometry: http://www.jmlr.org/papers/volume6/lafferty05a/lafferty05a.pdf

Answer 7

I also found this paper which gives a topological characterization of a classification neural network to be intuitive :

https://arxiv.org/abs/2008.13697

They defined data as a topological space and defined labels as closed subsets of this space. They also defined a notion of "separable data" which is topologically equivalent to "correctly classifying the data". Urysohn lemma is then used to prove that the data, considered as a topological space, can be "separated", if you can find embedded disks that are mutually disjoint and separate the labeled parts of the data (after being mapped by the neural network to the final space).

This can be seen with the disentanglement figure at the end of the paper:

which shows how the above neural network acts on the input topological space, deforms it in order to achieve a given classification task (in this case separating the two labeled linked spaces). Some hints to the general position theorem are also made there. There are also some links to "topological moves" in the paper :a neural network acts on the input topological space by a sequence of topological moves (similar to Reidemeister moves in knot theory or other moves in topology) in order to achieve the given task : which is deforming the input space to the final space, in this case the Voronoi diagram that represents the classes of the input data, such that every labeled region in the input space maps to the correct cell in the final Voronoi diagram.

To me, from this perspective, a classification neural network can be interpreted as if it is acting on the input space as a "discrete colored homotopy" : discrete because of the layers (time is the index of the layer) and colored because the class label which can be considered as "coloring" on the input space.

Answer 8

Since you mentioned homotopies, you might find interest in A homotopy training algorithm for fully connected neural networks. I haven't read the paper, but from the abstract I gather that the neural network (which is a function) is first trained and becomes $f(x)$. Then its architecture's complexity is increased and trained some more. The complexity increase (and training) is done multiple times until the final model, which is $g(x)$. The function $h$ from $f$ to $g$ is a homotopy.