Given a parametric family of distributions $\{p_\theta\mid\theta \in \Theta\}$, one can show that under some regularity conditions, the following approximation is valid

$$\operatorname{KL}(p_\theta\parallel p_{\theta + d \theta}) = d \theta^TF(\theta) \, d\theta + \mathcal O(\|d\theta\|^3), $$ where $$F(\theta)_{ij} := \mathbb E_{x \sim p_\theta}\left[\frac{\partial^2}{\partial \theta_i \, \partial \theta_j} \log(p_\theta(x))\right] $$ is the Fisher information matrix of $p_\theta$. A very rough sketch of the proof can be found on wikipedia.

Question 1

Is there such an approximation formula for the Wassertein distance or other measures of discrepancy between probability distributions ?

Question 2

Same question, specialized to $f$-divergences (of which KL is a particular case).

  • $\begingroup$ Talking to a colleague, it appears that Infinitesimally, $W_2$ at a measure $p_\theta$ matches at 2nd order the negative Sobolev space $H^{-1}(dp_\theta)$. This is explained in section 5.5.2 of Santambrogio OTAM book. $\endgroup$ – dohmatob Aug 11 '18 at 13:30
  • $\begingroup$ Also, Amari's works (e.g fluid.ippt.gov.pl/bulletin/%2858-1%29183.pdf) also appear to be good reference on the 2nd question (the answer is mostly positive). $\endgroup$ – dohmatob Aug 12 '18 at 1:15

With regard to your first question, one such result shows that under some conditions on $q$, the following inequality holds for all p:

$$W_2(p,q) \leq \sqrt{\frac{ KL(p\parallel q)}\rho}$$

This is a so called Talagrand$(\rho)$ inequality, which holds whenever a suitable log-Sobolev inequality holds for $q$. For more information, the paper of Otto and Villani [1] is a good reference. This is all contained in Villani's book on optimal transport as well. This also contains many other inequalities between the various distances and divergences, so it is a good reference. However, due to the work of Otto, the Wasserstein 2 metric induces a formal Riemannian metric on the space of densities, and its geometry can be quite different from that of the Fisher-Rao metric.

As to your second question, the Fisher metric is unique in that it is the quadratic term of the Taylor series for any $f$-divergence. For more information on this phenomena, see [2].

[1] Otto, F., & Villani, C. (2000). Generalization of an inequality by Talagrand and links with the logarithmic Sobolev inequality. Journal of Functional Analysis, 173(2), 361-400.

[2] The Fisher Metric Will Not Be Deformed https://golem.ph.utexas.edu/category/2018/05/the_fisher_metric_will_not_be.html

| cite | improve this answer | |
  • $\begingroup$ Thanks for the response. The Otto et Villani paper is on point. It also contains a strange pearl (a consequence of Talagrand's inequality): concentration of Gaussian measure for the $t$-blowup $B_t$ of a Borel set $B$ in $\mathbb R^d$, namely $\gamma_d(B_t) \ge 1-\exp(-\frac{1}{2}(t-\sqrt{1\log(1/\gamma_d(B))})^2)$. $\endgroup$ – dohmatob Aug 10 '18 at 6:31
  • $\begingroup$ Also, Amari's works (e.g fluid.ippt.gov.pl/bulletin/%2858-1%29183.pdf) also appear to be good reference on the 2nd question (the answer is mostly positive). $\endgroup$ – dohmatob Aug 12 '18 at 1:15
  • 1
    $\begingroup$ As a token of gratitute, I should mention that the Otto et Villani reference you mentioned in your answer got me deep-diving into the subject of measure concentration, with a eye towards machine-learning applications. Here is my first produce arxiv.org/pdf/1810.04065.pdf $\endgroup$ – dohmatob Oct 19 '18 at 10:10
  • $\begingroup$ Awesome! I'm glad that reference was helpful for your work. $\endgroup$ – Gabe K Oct 19 '18 at 14:46

A natural field here is Wasserstein information geometry.

See Wuchen Li, Guido Montufar: Natural gradient via optimal transport


For related applications see my talk


| cite | improve this answer | |
  • 2
    $\begingroup$ 1) You should have specified that you link to your own works. 2) What is the OP supposed to do with the links? What if the second one dies? How do they answer the question? $\endgroup$ – Alex M. Apr 22 '19 at 6:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.