# How useful is differential geometry and topology to deep learning?

After seeing this article https://www.quantamagazine.org/an-idea-from-physics-helps-ai-see-in-higher-dimensions-20200109/ I wanted to ask myself how useful of an endeavor would it be if one goes through the process of learning differential geometry to deep learning? Perhaps there are other tools as well?

Can concepts such as homotopy be 'appropriately' encodable in deep learning? I feel notions of homotopy invariance if properly defined could be much more valuable than the simple attempt in quantamagazine article. A basic attempt (away from deep learning framework) is in https://arxiv.org/abs/1408.2584.

• Day-to-day application of machine learning generally uses simple statistical models, such as GLMs. Most of the work is in modeling and cleaning data and figuring out how to communicate results to non-technical people (whether through reports and data viz or embedding in a production application). So I'd expect differential geometry/topology are not immediately useful in industry jobs outside of big tech companies' research labs. – Neal Jan 11 '20 at 17:47
• @Neal I doubt it will still be that way in the future if progress is made. Right now it is not the case because not much is known to architecture the divide and conquer process. If properly framed ML (as it is envisioned today) could benefit from modern mathematics (we can always sell the product nicely to lay people with nice words as currently done with quantum computing). – VS. Jan 11 '20 at 19:43
• I don't think there is a definitive answer to your question at this point. Presumably the answer will eventually be a strong "yes, very useful". But as far as I am aware, at present it sounds like "maybe in some special cases" is more appropriate. – Ryan Budney Jan 11 '20 at 20:37
• I suspect there's at least one quality PhD dissertation in ML that will be written about employing differential geometry in ML. Even if the topic is of interest to only academia for quite a long time, it would still set you up nicely for a career in both academia and industry. – Betterthan Kwora Jan 12 '20 at 2:08
• Differential geometry can take a long time to learn but the benefits are that you can consider curvature for manifolds of arbitrary dimension, which is worth the effort imo. – Hollis Williams Jan 13 '20 at 7:58

A "roadmap type" introduction is given by Roger Grosse in Differential geometry for machine learning.

Differential geometry is all about constructing things which are independent of the representation. You treat the space of objects (e.g. distributions) as a manifold, and describe your algorithm in terms of things that are intrinsic to the manifold itself. While you ultimately need to use some coordinate system to do the actual computations, the higher-level abstractions make it easier to check that the objects you're working with are intrinsically meaningful. This roadmap is intended to highlight some examples of models and algorithms from machine learning which can be interpreted in terms of differential geometry. Most of the content in this roadmap belongs to information geometry, the study of manifolds of probability distributions. The best reference on this topic is probably Amari and Nagaoka's Methods of Information Geometry.

• One should note that this kind of application of differential geometry to ML is different to the one described in the quanta article. The former is based on the infinite-dimensional manifold of (probability) distributions while the work of Cohen et al (arxiv.org/abs/1902.04615) is concerned with data and features that life on a finite-dimensional manifold, e.g. vector fields on a sphere. – Tobias Diez Jan 11 '20 at 22:53
• Also is it relevant to deep learning mathematical architecture and intuition? – VS. Jan 12 '20 at 18:25
• This answer does not seem to be about deep learning. – VS. Jan 13 '20 at 11:04

https://arxiv.org/abs/1805.10451

Although what they say is by no means new - Coifman had had this point of view for the last thirty years at least.

Topological data analysis is already pretty popular in "data science", so much so that there are companies built on this idea. See: 1). https://en.wikipedia.org/wiki/Topological_data_analysis 2).https://web.stanford.edu/class/archive/ee/ee392n/ee392n.1146/lecture/may13/EE392n_TDA_online.pdf 3). https://towardsdatascience.com/a-concrete-application-of-topological-data-analysis-86b89aa27586

• TDA as of now how it stands does not appear very interesting for many fields such as NLP or vision let alone invariance detection in general. – VS. Jan 11 '20 at 19:36
• @VS I am not sure what the success stories of TDA are. I am sure that's due to my own ignorance. – Igor Rivin Jan 11 '20 at 19:44
• @IgorRivin I think TDA as of now is too simple to have any applications useful to industry. The right way seems to be deep learning (which is some glorified non-uniform divide and conquer by design of architecture) with some elaborate context specific structural information and intuition from mathematics that specifies the architecture. That is why I think differential geometry as of how it is applied now is not the end of story. – VS. Jan 11 '20 at 19:45
• Well link #2 is presentation by a industrial firm on their use of TDA. So atleast they claim it is useful – Piyush Grover Jan 11 '20 at 20:28
• @ Piyush Grove Link #2 ls not by an industrial firm. It is by Ayasdi, which is a company which produces and sells Topological Data Analysis software..In your parlance, one of the companies built on this idea. – Mark L. Stone Jan 12 '20 at 1:38

Taco Cohen has written some papers that use Differential Geometry, Topology, Guage theory, etc. in Machine learning.

• This is about deep learning. – VS. Jan 13 '20 at 11:04
• Yes, Cohen does apply it on deep learning. – hrkrshnn Jan 13 '20 at 11:18
• That is what I mean contrasting CarloBeenakker's answer. – VS. Jan 13 '20 at 11:20

Colah gives a very interesting perspective about deep learning and neural networks in the context of topology. He discusses the "Manifold Hypothesis" which, in short, tries to explain why deep learning is so effective. To read more about the Manifold Hypothesis, Goodfellow has a chapter on it.

On another note, there are interesting articles being published in the context of topological analysis of some "image spaces". For example, Carlsson et al.:

In this study we concentrate on qualitative topological analysis of the local behavior of the space of natural images. To this end, we use a space of 3 by 3 high-contrast patches ℳ. We develop a theoretical model for the high-density 2-dimensional submanifold of ℳ showing that it has the topology of the Klein bottle.

I think learning basic topological ideas and especially all the work that has already been done on the connection of data analysis and topology can be very fruitful (Mind that I am no expert though).

• Is "manifold hypothesis" supposed to be glossed as "low-dimensional subspace hypothesis", or is smoothness really an aspect of it? This reminds me of the "T" in "TQFT", which stands for "topological" but typically means "smooth". – Tim Campion Feb 11 at 18:46
• As far as I can remember it is usually only meant as "low-dimensional subspace hypothesis", yes. – horropie Feb 11 at 19:05

Heat kernels are based on ideas from differential geometry: http://www.jmlr.org/papers/volume6/lafferty05a/lafferty05a.pdf

I also found this paper which gives a topological characterization of a classification neural network to be intuitive :

https://arxiv.org/abs/2008.13697

They defined data as a topological space and defined labels as closed subsets of this space. They also defined a notion of "separable data" which is topologically equivalent to "correctly classifying the data". Urysohn lemma is then used to prove that the data, considered as a topological space, can be "separated", if you can find embedded disks that are mutually disjoint and separate the labeled parts of the data (after being mapped by the neural network to the final space).

This can be seen with the disentanglement figure at the end of the paper:

which shows how the above neural network acts on the input topological space, deforms it in order to achieve a given classification task (in this case separating the two labeled linked spaces). Some hints to the general position theorem are also made there. There are also some links to "topological moves" in the paper :a neural network acts on the input topological space by a sequence of topological moves (similar to Reidemeister moves in knot theory or other moves in topology) in order to achieve the given task : which is deforming the input space to the final space, in this case the Voronoi diagram that represents the classes of the input data, such that every labeled region in the input space maps to the correct cell in the final Voronoi diagram.

To me, from this perspective, a classification neural network can be interpreted as if it is acting on the input space as a "discrete colored homotopy" : discrete because of the layers (time is the index of the layer) and colored because the class label which can be considered as "coloring" on the input space.

Since you mentioned homotopies, you might find interest in A homotopy training algorithm for fully connected neural networks. I haven't read the paper, but from the abstract I gather that the neural network (which is a function) is first trained and becomes $$f(x)$$. Then its architecture's complexity is increased and trained some more. The complexity increase (and training) is done multiple times until the final model, which is $$g(x)$$. The function $$h$$ from $$f$$ to $$g$$ is a homotopy.