22
$\begingroup$

After seeing this article https://www.quantamagazine.org/an-idea-from-physics-helps-ai-see-in-higher-dimensions-20200109/ I wanted to ask myself how useful of an endeavor would it be if one goes through the process of learning differential geometry to deep learning? Perhaps there are other tools as well?

Can concepts such as homotopy be 'appropriately' encodable in deep learning? I feel notions of homotopy invariance if properly defined could be much more valuable than the simple attempt in quantamagazine article. A basic attempt (away from deep learning framework) is in https://arxiv.org/abs/1408.2584.

$\endgroup$
  • 9
    $\begingroup$ Day-to-day application of machine learning generally uses simple statistical models, such as GLMs. Most of the work is in modeling and cleaning data and figuring out how to communicate results to non-technical people (whether through reports and data viz or embedding in a production application). So I'd expect differential geometry/topology are not immediately useful in industry jobs outside of big tech companies' research labs. $\endgroup$ – Neal Jan 11 at 17:47
  • 1
    $\begingroup$ @Neal I doubt it will still be that way in the future if progress is made. Right now it is not the case because not much is known to architecture the divide and conquer process. If properly framed ML (as it is envisioned today) could benefit from modern mathematics (we can always sell the product nicely to lay people with nice words as currently done with quantum computing). $\endgroup$ – VS. Jan 11 at 19:43
  • 3
    $\begingroup$ I don't think there is a definitive answer to your question at this point. Presumably the answer will eventually be a strong "yes, very useful". But as far as I am aware, at present it sounds like "maybe in some special cases" is more appropriate. $\endgroup$ – Ryan Budney Jan 11 at 20:37
  • 1
    $\begingroup$ Learning differential geometry certainly can't hurt, as the answers below show. The only real downside is opportunity cost: depending on your objective and your current knowledge, there could easily be another subject that's even more helpful. Can you say more about your goal? (E.g., practical applications, personal intuition, new research results, etc.) $\endgroup$ – Martin M. W. Jan 12 at 1:05
  • 2
    $\begingroup$ I suspect there's at least one quality PhD dissertation in ML that will be written about employing differential geometry in ML. Even if the topic is of interest to only academia for quite a long time, it would still set you up nicely for a career in both academia and industry. $\endgroup$ – Betterthan Kwora Jan 12 at 2:08
17
$\begingroup$

A "roadmap type" introduction is given by Roger Grosse in Differential geometry for machine learning.

Differential geometry is all about constructing things which are independent of the representation. You treat the space of objects (e.g. distributions) as a manifold, and describe your algorithm in terms of things that are intrinsic to the manifold itself. While you ultimately need to use some coordinate system to do the actual computations, the higher-level abstractions make it easier to check that the objects you're working with are intrinsically meaningful. This roadmap is intended to highlight some examples of models and algorithms from machine learning which can be interpreted in terms of differential geometry. Most of the content in this roadmap belongs to information geometry, the study of manifolds of probability distributions. The best reference on this topic is probably Amari and Nagaoka's Methods of Information Geometry.

$\endgroup$
  • 1
    $\begingroup$ One should note that this kind of application of differential geometry to ML is different to the one described in the quanta article. The former is based on the infinite-dimensional manifold of (probability) distributions while the work of Cohen et al (arxiv.org/abs/1902.04615) is concerned with data and features that life on a finite-dimensional manifold, e.g. vector fields on a sphere. $\endgroup$ – Tobias Diez Jan 11 at 22:53
  • $\begingroup$ Also is it relevant to deep learning mathematical architecture and intuition? $\endgroup$ – VS. Jan 12 at 18:25
  • $\begingroup$ This answer does not seem to be about deep learning. $\endgroup$ – VS. Jan 13 at 11:04
14
$\begingroup$

You can read this paper:

https://arxiv.org/abs/1805.10451

Although what they say is by no means new - Coifman had had this point of view for the last thirty years at least.

$\endgroup$
7
$\begingroup$

Topological data analysis is already pretty popular in "data science", so much so that there are companies built on this idea. See: 1). https://en.wikipedia.org/wiki/Topological_data_analysis 2).https://web.stanford.edu/class/archive/ee/ee392n/ee392n.1146/lecture/may13/EE392n_TDA_online.pdf 3). https://towardsdatascience.com/a-concrete-application-of-topological-data-analysis-86b89aa27586

$\endgroup$
  • $\begingroup$ TDA as of now how it stands does not appear very interesting for many fields such as NLP or vision let alone invariance detection in general. $\endgroup$ – VS. Jan 11 at 19:36
  • 1
    $\begingroup$ @VS I am not sure what the success stories of TDA are. I am sure that's due to my own ignorance. $\endgroup$ – Igor Rivin Jan 11 at 19:44
  • $\begingroup$ @IgorRivin I think TDA as of now is too simple to have any applications useful to industry. The right way seems to be deep learning (which is some glorified non-uniform divide and conquer by design of architecture) with some elaborate context specific structural information and intuition from mathematics that specifies the architecture. That is why I think differential geometry as of how it is applied now is not the end of story. $\endgroup$ – VS. Jan 11 at 19:45
  • $\begingroup$ Well link #2 is presentation by a industrial firm on their use of TDA. So atleast they claim it is useful $\endgroup$ – Piyush Grover Jan 11 at 20:28
  • 3
    $\begingroup$ @ Piyush Grove Link #2 ls not by an industrial firm. It is by Ayasdi, which is a company which produces and sells Topological Data Analysis software..In your parlance, one of the companies built on this idea. $\endgroup$ – Mark L. Stone Jan 12 at 1:38
5
$\begingroup$

Taco Cohen has written some papers that use Differential Geometry, Topology, Guage theory, etc. in Machine learning.

$\endgroup$
  • $\begingroup$ This is about deep learning. $\endgroup$ – VS. Jan 13 at 11:04
  • $\begingroup$ Yes, Cohen does apply it on deep learning. $\endgroup$ – jdoicj Jan 13 at 11:18
  • $\begingroup$ That is what I mean contrasting CarloBeenakker's answer. $\endgroup$ – VS. Jan 13 at 11:20
4
$\begingroup$

Heat kernels are based on ideas from differential geometry: http://www.jmlr.org/papers/volume6/lafferty05a/lafferty05a.pdf

$\endgroup$
2
$\begingroup$

Colah gives a very interesting perspective about deep learning and neural networks in the context of topology. He discusses the "Manifold Hypothesis" which, in short, tries to explain why deep learning is so effective. To read more about the Manifold Hypothesis, Goodfellow has a chapter on it.

On another note, there are interesting articles being published in the context of topological analysis of some "image spaces". For example, Carlsson et al.:

In this study we concentrate on qualitative topological analysis of the local behavior of the space of natural images. To this end, we use a space of 3 by 3 high-contrast patches ℳ. We develop a theoretical model for the high-density 2-dimensional submanifold of ℳ showing that it has the topology of the Klein bottle.

I think learning basic topological ideas and especially all the work that has already been done on the connection of data analysis and topology can be very fruitful (Mind that I am no expert though).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.