39
$\begingroup$

The space of probability measures could be viewed as an infinite-dimensional manifold, equipped with two possible types of metrics — (1) Wasserstein and (2) Fisher-Rao. Metric (1) is connected with optimal transport, and metric (2) is connected with information geometry.

Question: What are the connections between these two metrics? We know that the Fisher-Rao metric is characterized by the Fisher information matrix, but what is the corresponding characterization for the Wasserstein metric? Any references are much appreciated.

Improve this question
$\endgroup$
9
  • 11
    $\begingroup$ It is quite bold to consider the Wasserstein metric as a Riemannian metric. There is no genuine (infinite dimensional) manifold structure involved, let alone a truly Riemannian structure. You do have tangent cones but only at some points are they tangent spaces. Some of these problems disappear when you restrict to absolutely continuous measures, but then this is far from being a closed set. $\endgroup$
    – Benoît Kloeckner
    Commented Jul 30, 2017 at 8:32
  • 1
    $\begingroup$ @BenoîtKloeckner in principle one could do something like look at a generalised smooth structure (for instance, diffeological, Fröhlicher,...), but I don't know the technicalities enough to say it would work. $\endgroup$
    – David Roberts ♦
    Commented Jul 31, 2017 at 1:11
  • 3
    $\begingroup$ You may find the following paper interesting: arxiv.org/abs/1208.0434 $\endgroup$
    – Suvrit
    Commented Jul 31, 2017 at 1:28
  • 15
    $\begingroup$ @Minkov , You can give a connections between canonical metrics(Kahler-Einstein geometry) and random point processes on a complex algebraic variety . When the variety X has positive Kodaira dimension, this leads to a probabilistic construction of the canonical metric on X introduced by Song-Tian-Tsuji. see my paper hal.archives-ouvertes.fr/hal-01413746 . Also K-stability of Tian and Donaldson can introduced by Gibbs probability measure and its connections with discrete optimal transport theory, see recent papers of Berman arxiv.org/abs/1401.8264 $\endgroup$
    – user21574
    Commented Jul 31, 2017 at 5:07
  • 2
    $\begingroup$ This would be a better question without the word "Riemannian". $\endgroup$
    – user44143
    Commented Jul 31, 2017 at 13:29

3 Answers 3

Reset to default
18
$\begingroup$

In response to the critical comments below I revised my answer. Hope this is more helpful!

(1) Two kinds of metrics are defined on generally different spaces.

It is not fair to compare these two metric since the Fisher-Rao is defined for probability densities defined on space $(X,\mu)$, the elements of concern are in space $M(X,\mu)$; while the Wasserstein is defined directly on probability measures on $X$ with/without densities, the elements of concern are in space $M(X)$. Even if we step back and say we concern those probability measures with absolutely continuous densities and the manifold they defined, it is readily observed that

Fisher metric provides a negative constant sectional curvature while Wasserstein metric is flat [3]

Negative curvature is favorable due to various comparison theorems in Riemanian geometry.(See a different opinion from Kloeckner's comment below)a strictly negative curvature can be regarded as convexity of the family of measures under concern making the minimizer of KL divergence unique. The reflection of this point in information geometry is the natural relation between KL divergence and Fisher-Rao metric, which correspondence Wasserstein distance does not have.[1] Meanwhile the Wasserstein metric has a natural connection with optimal transportation theory which Fisher-Rao metric may not provide.

(2)What are the connections between these two metrics? They both somehow characterize the dependence between two distributions using geodesic distance w.r.t. metrics.

(2.1)Dependence characterization using metric geodesic distance

  • (i)Fisher-Rao distance between probability densities measures is equivalent to the geodesic distance in sense of the 'correct' geodesic distance characterized by Cencov's theorem[6].

  • (ii)Wasserstein distance between probability measures is equivalent to the most correlated copula associated with these measures. [2]

Therefore the Wasserstein metric may reflect the dependence(entropy of data-generating process) better than Fisher-Rao in strongly dependent case[2]. This statement can be made precise using Newman's language[4] describing the bounded-Lipshictz dependence mentioned in [5].

(2.2)Fisher-Rao depends on the underlying manifold.

However, Fisher-Rao metric also depends on the embedding manifold and have parametric and nonparametric version. For example, when the underlying manifold is all Gaussian measures with varying means, then the Rao-Fisher metric is simply a linear metric when measuring the geodesic distance between two points on this manifold; when the underlying manifold is all measures with finite second moments, then the Rao-Fisher metric is not linear anymore. So the discussion is also affected by what underlying manifold we have in mind.

(2.3)Wasserstein metric does not depend on the underlying manifold by its definition since it must take variation over all measures with prescribed margins.

(3) Cencov's Theorem

Very loosely, Fisher-Rao metric, due to Cencov theorem [6] that Fisher-Rao metric is the 'correct' metric to use when the transition mappings are selected to be Markov morphisms.

...proved that the Fisher-Rao metric is the only metric that is invariant under mappings referred to as congruent embeddings by Markov morphisms.[7]

(4)What is the corresponding characterization for the Wasserstein metric?

As far as I know there is no a characterization for general $L^p$ Wasserstein metric, but in some cases like $L^2$, the minimizer to Wasserstein metric is derived [8] as optimal couplings. These characterizations are also useful in reality [2]. As OP mentioned, the minimizer of Fisher-Rao metric can be characterized using FIsher information matrix and in exponential family these are MLEs.

Reference

[1]Amari, Shun-ichi. "Divergence function, information monotonicity and information geometry." Workshop on Information Theoretic Methods in Science and Engineering (WITMSE). 2009.

[2]Marti, Gautier, et al. "Optimal transport vs. Fisher-Rao distance between copulas for clustering multivariate time series." Statistical Signal Processing Workshop (SSP), 2016 IEEE. IEEE, 2016.

[3]Barbaresco, Frédéric. "Geometric radar processing based on Fréchet distance: information geometry versus optimal transport theory." Radar Symposium (IRS), 2011 Proceedings International. IEEE, 2011.

[4]Newman, Morris. "Periodicity modulo m and divisibility properties of the partition function." Transactions of the American Mathematical Society 97.2 (1960): 225-236.

[5]Bulinski, A. V. and Vronski, M. A. (1996). Statistical variant of the central limit theorem for associated random elds, Fundam. Prikl. Mat., 2, 4, pp. 999{1018 (in Russian).

[6]Cencov, Nikolai Nikolaevich. Statistical decision rules and optimal inference. No. 53. American Mathematical Soc., 2000.

[7]Peter, Adrian, and Anand Rangarajan. "Shape analysis using the Fisher-Rao Riemannian metric: Unifying shape representation and deformation." Biomedical Imaging: Nano to Macro, 2006. 3rd IEEE International Symposium on. IEEE, 2006.

[8]Rüschendorf, L., & Rachev, S. T. (1990). A characterization of random variables with minimum L2-distance. Journal of Multivariate Analysis, 32(1), 48-54.

Improve this answer
$\endgroup$
8
  • 4
    $\begingroup$ There are many issues with this answer. First, I don't see in what sense Wasserstein metric could be considered flat (except if the underlying space is the line): it contains an isometric copy of the underlying manifold as a totally geodesic subspace, so there is at least as much non-zero curvature in the Wassertein space than in the underlying manifold. In fact there is more, as Wasserstein spaces of manifolds (of dimension 2 or more) have positive curvature properties at some points and directions... $\endgroup$
    – Benoît Kloeckner
    Commented Aug 3, 2017 at 19:46
  • 3
    $\begingroup$ ..."Fisher-Rao only captures the shortest (geodesic) distance direction between two points(probabilities)" makes little sense to me, as any metric is precisely about defining distance between points; and Wasserstein distance is geodesic (but I guess with different geodesic from those of the Fisher-Rao metric, which I don't know about). Several other similar claims make no more sense to me, for the same reason ("Fisher-Rao distance between probability densities measures is equivalent to the geodesic distance" - geodesics according to what metric?)... $\endgroup$
    – Benoît Kloeckner
    Commented Aug 3, 2017 at 19:49
  • 4
    $\begingroup$ Last, "Negative curvature is always favorable due to various comparison theorems in Riemanian geometry" is pretty dubious: flatness is very restrictive in Riemannian geometry (e.g. flat Riemannian 3-manifold where classified maybe a century ago, while negatively curved one are somewhat well understood thanks to the geometrization conjecture of Thurston, proved by Perelman); and there are comparison theorem or both upper and lower bounds on curvature. $\endgroup$
    – Benoît Kloeckner
    Commented Aug 3, 2017 at 19:51
  • 1
    $\begingroup$ @BenoîtKloeckner 1)"...flat" is a word-by-word claim from a paper I explicitly cited and believed, a detail argument is there, it is impossible and meaningless for me to reproduce the argument 2)Under RF metric Wasserstein usually provide a different geodesic that is not geodesic under RF. However RF is the 'correct' metric to use due to Cencov theorem, see my update. 3)You're more than welcome to give another answer that clarifies this, my point of saying this is that negative curvature is intepreted as log convexity of the densities consisting of the probability manifold(e.g. exp family $\endgroup$
    – Henry.L
    Commented Aug 3, 2017 at 22:22
  • $\begingroup$ @BenoîtKloeckner In fact, 2) is also explained in the paper [2] which illustrated why FR captures the dependence better using example of time series, than Wasserstein if you read it carefully. The [5] I cited above contains a very clear explanation about why covariance can be treated as a measure of dependence and hence allowed us to view Newman-type inequalities as a control on curvature. Thanks for your historical comment. $\endgroup$
    – Henry.L
    Commented Aug 3, 2017 at 22:29
12
$\begingroup$

Edit (June 2022): Jun Zhang and I wrote a survey paper on some interactions between these two fields which expands on what I mentioned here. You can find the paper at the following link: https://arxiv.org/abs/2206.14791

Original answer: This is an excellent question, but I don't think there is a clean and simple answer. In general, these two metrics reflect different things for probability measures and they interact in interesting ways.

Before getting too much into the weeds, I'll mention a paper that I found extremely helpful to my own understanding of the relationship.

Khesin, B., Lenells, J., Misiołek, G., & Preston, S. C. (2013). Geometry of diffeomorphism groups, complete integrability and geometric statistics. Geometric and Functional Analysis, 23(1), 334-366. http://www.math.toronto.edu/khesin/papers/H1-gafa.pdf

With that out of the way, I'll mention a few general phenomena:

  1. The Fisher-Rao metric is a canonical Riemannian metric on a parametrized statistical manifold. Generally, one does not do optimal transport of a finite dimensional parametrized family of distributions, so to make any comparisons, it is instructive to consider the infinite-dimensional non-parametric version of a statistical manifold. The paper that I cited above does this extremely clearly and explains how to extend the Fisher-Rao metric to the infinite-dimensional setting as a particular $H^1$ metric. There have been other attempts to study non-parametrized statistical manifolds, but these generally involve Orlisz spaces and a lot of seemingly ad hoc machinery. To be honest, I am not aware of what new mathematical insights this more complicated theory yields. Perhaps someone with more experience can fill me in.

  2. It is worth noting that the space of probability measures on a metric space $X$, with the distance induced by the Wasserstein 2 metric is not, in general, a Riemmanian manifold. However, once some extra regularity has been assumed, Otto did introduce a formal Riemannian structure on this space. This is an important notion, but it is formal. The curvature of this structure depends on the underlying metric of $X$. In particular, $(P(X),\mathcal{W}_2)$ has non-negative sectional curvature (in the sense of Topogonov) if and only if $X$ does. For more information on this, the following paper of John Lott goes into a lot more detail. https://math.berkeley.edu/~lott/cmp.pdf

  3. By contrast, the $H^1$ metric on probability distributions does not depend on the metric, which is somehow a consequence of the fact that the entropy is diffeomorphism invariant. As shown in the first paper I cited, the space of probability densities with $H^1$ metric is isometric to the positive part of the sphere in Hilbert space. As such, it has constant positive sectional curvature. Furthermore, this makes distances and geodesics in the $H^1$ metric much easier to compute.

  4. I've somewhat been dancing around the issue, but the Wasserstein distance and $H^1$ metric are defined for different spaces. Furthermore, if one considers the topologies induced by the corresponding metric, these are different. In general, the Wasserstein distance is defined for more general probability measures whereas the information-geometric notions generally require probability densities.

  5. There are many important theorems that relate the entropic notions of statistical distance to the optimal transport notions. For instance, the Talagrand inequality shows that the squared Wasserstein distance is bounded by the relative entropy (under assumptions). For more information, the following paper of Otto and Villani is a great reference. http://cedricvillani.org/wp-content/uploads/2012/08/014.OV-Talagrand.pdf

    In general, we can expect to have inequalities that control the Wasserstein distance by the entropic distances. However, we don't really expect to have inequalities going the other way. There are multiple ways to think of this, but it boils down to the fact that the entropic distances generally require more regularity in their definition and we don't expect to be able to bound higher derivatives by lower ones.

  6. It seems that information geometry does not only use the Fisher metric, but also many other "divergences," which are distances where you drop the requirements of symmetry and the triangle inequality. There is a theorem that shows that a very general class of divergences (the $f$-divergences) induce the Fisher-Rao metric, which is why it is a canonical metric. However, it is often worthwhile to consider the full divergence, not just the metric. To be honest, I am not completely sold on the idea of just defining more and more general divergences. It seems to me that one should have a good reason to consider a particular divergence and an application in mind.

    For practical purposes, the choice to use the Wasserstein metric vs. an information geometric divergence really depends on the structure of your problem. Should the distance incorporate the underlying metric of your space or is it just a matter of how far they are from an information theory point of view?

  7. One can see what happens if you consider a linear combination of the Wasserstein and Fisher-Rao metrics. Two separate groups of people have researched this independently in the last few years, and have some interesting results on the associated metric space. Here are links to two of the papers in this new formulation. (https://arxiv.org/pdf/1508.07941.pdf and https://arxiv.org/pdf/1505.07746.pdf).

These new metrics are useful for studying reaction-diffusion equations, where the information geometry induces reaction while the Wasserstein geometry induces diffusion. It takes some work to make the preceding sentence precise, but it turns out to be a good general principle.

From a meta perspective, information geometry has a much smaller community within the math world than optimal transport does. Part of this is that it is a much younger field, but I also think it currently lacks the sort of punchlines that optimal transport has. In order to motivate a mathematician to consider optimal transport, one can easily point to some of the big theorems and conjectures (e.g. "Nearly round spheres look convex"). I'm not aware of similar results in information geometry. As a final note, classical information geometry is largely based off of affine differential geometry, which is distinct from Riemannian geometry. There seems to have been a small renaissance of affine differential geometry in the past 15 years. One hopes that some of these breakthroughs can be used to develop and stimulate interest in information geometry.

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ This is excellent. By the affine geometry comment, are you referring to the dual connections as opposed to the Levi-Civita connection? $\endgroup$
    – S.Surace
    Commented Sep 26, 2018 at 7:47
  • 1
    $\begingroup$ that's probably fodder for a totally new MO question :). Another comment regarding the curvature of Fisher-Rao: I guess what confuses people (including myself) is that the Levi-Civita connection of pullbacks of the Fisher-Rao metric along embeddings of finite-dimensional statistical models are often negatively curved. The canonical example is the space of 1d Gaussian distributions, which has a hyperbolic Fisher metric. $\endgroup$
    – S.Surace
    Commented Sep 26, 2018 at 15:40
  • 1
    $\begingroup$ Right, so this is something I've thought a little bit about, because I was initially very confused as well. The first thing is that the result of Khesin et. al. applies to non-parametric statistical manifold where the base space $X$ has finite measure (so I told a small lie). If the measure of $X$ is infinite, then the corresponding $H^1$ space is actually flat. However, this is somewhat orthogonal to the actual point. A finite dimensional statistical manifold is an infinite codimensional subset of this giant non-parametric space. (Continued in the next comment) $\endgroup$
    – Gabe K
    Commented Sep 26, 2018 at 15:47
  • 1
    $\begingroup$ There is no reason to expect that the curvature of a submanifold with high codimension says anything about the curvature of the ambient space. In fact, Nash's embedding theorem (for which Le proved an information geometric version) implies that any Riemannian manifold can be isometrically embedded in Euclidean space. As such, we should expect to find finite dimensional statistical manifolds with negative curvature, and some of the important examples bear this out. $\endgroup$
    – Gabe K
    Commented Sep 26, 2018 at 15:51
  • 1
    $\begingroup$ @GabeK Nice to see you on MO :) $\endgroup$
    – Henry.L
    Commented Mar 10, 2019 at 14:46
8
$\begingroup$

Just a quick follow-up: Very recently (well, actually in 2015) three teams came up independently and almost simultaneously with the same construction of a new "optimal-transport-like" distance on the space of Radon measures $\mathcal M^+$, which somehow interpolates continuously between Wasserstein and Fisher-Rao. This distance now goes by the name of Wasserstein-Fisher-Rao (WFR) metrics, sometimes also Hellinger-Kantorovich (HK) distance, and gave rise to a whole new topics generally referred to as unbalanced optimal transport. As pointed out by @GabeK in his excellent answer the Wasserstein and Fisher-Rao structures interact in an interesting way and lead to unsuspected behaviour (well, at least for me). The underlying structure possesses several rich and geometric underlying formulations. In a nutshell, the WFR distance can be seen as an infimal-convolution of Wasserstein and Fisher-Rao distances.

For recent developments see the citations of the original trhee papers below (I belong to the first team)

[1] Kondratyev, S., Monsaingeon, L., & Vorotnikov, D. (2016). A new optimal transport distance on the space of finite Radon measures. Advances in Differential Equations, 21(11/12), 1117-1164.

[2] Chizat, L., Peyré, G., Schmitzer, B., & Vialard, F. X. (2018). An interpolating distance between optimal transport and Fisher–Rao metrics. Foundations of Computational Mathematics, 18(1), 1-44.

[3] Liero, M., Mielke, A., & Savaré, G. (2018). Optimal entropy-transport problems and a new Hellinger–Kantorovich distance between positive measures. Inventiones mathematicae, 211(3), 969-1117.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .