8
$\begingroup$

Given a parametric family of distributions $\{p_\theta\mid\theta \in \Theta\}$, one can show that under some regularity conditions, the following approximation is valid

$$\operatorname{KL}(p_\theta\parallel p_{\theta + d \theta}) = d \theta^TF(\theta) \, d\theta + \mathcal O(\|d\theta\|^3), $$ where $$F(\theta)_{ij} := \mathbb E_{x \sim p_\theta}\left[\frac{\partial^2}{\partial \theta_i \, \partial \theta_j} \log(p_\theta(x))\right] $$ is the Fisher information matrix of $p_\theta$. A very rough sketch of the proof can be found on wikipedia.

Question 1

Is there such an approximation formula for the Wassertein distance or other measures of discrepancy between probability distributions ?

Question 2

Same question, specialized to $f$-divergences (of which KL is a particular case).

Improve this question
$\endgroup$
2
  • $\begingroup$ Talking to a colleague, it appears that Infinitesimally, $W_2$ at a measure $p_\theta$ matches at 2nd order the negative Sobolev space $H^{-1}(dp_\theta)$. This is explained in section 5.5.2 of Santambrogio OTAM book. $\endgroup$
    – dohmatob
    Commented Aug 11, 2018 at 13:30
  • $\begingroup$ Also, Amari's works (e.g fluid.ippt.gov.pl/bulletin/%2858-1%29183.pdf) also appear to be good reference on the 2nd question (the answer is mostly positive). $\endgroup$
    – dohmatob
    Commented Aug 12, 2018 at 1:15

2 Answers 2

Reset to default
3
$\begingroup$

With regard to your first question, one such result shows that under some conditions on $q$, the following inequality holds for all p:

$$W_2(p,q) \leq \sqrt{\frac{ KL(p\parallel q)}\rho}$$

This is a so called Talagrand$(\rho)$ inequality, which holds whenever a suitable log-Sobolev inequality holds for $q$. For more information, the paper of Otto and Villani [1] is a good reference. This is all contained in Villani's book on optimal transport as well. This also contains many other inequalities between the various distances and divergences, so it is a good reference. However, due to the work of Otto, the Wasserstein 2 metric induces a formal Riemannian metric on the space of densities, and its geometry can be quite different from that of the Fisher-Rao metric.

As to your second question, the Fisher metric is unique in that it is the quadratic term of the Taylor series for any $f$-divergence. For more information on this phenomena, see [2].

[1] Otto, F., & Villani, C. (2000). Generalization of an inequality by Talagrand and links with the logarithmic Sobolev inequality. Journal of Functional Analysis, 173(2), 361-400.

[2] The Fisher Metric Will Not Be Deformed https://golem.ph.utexas.edu/category/2018/05/the_fisher_metric_will_not_be.html

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the response. The Otto et Villani paper is on point. It also contains a strange pearl (a consequence of Talagrand's inequality): concentration of Gaussian measure for the $t$-blowup $B_t$ of a Borel set $B$ in $\mathbb R^d$, namely $\gamma_d(B_t) \ge 1-\exp(-\frac{1}{2}(t-\sqrt{1\log(1/\gamma_d(B))})^2)$. $\endgroup$
    – dohmatob
    Commented Aug 10, 2018 at 6:31
  • $\begingroup$ Also, Amari's works (e.g fluid.ippt.gov.pl/bulletin/%2858-1%29183.pdf) also appear to be good reference on the 2nd question (the answer is mostly positive). $\endgroup$
    – dohmatob
    Commented Aug 12, 2018 at 1:15
  • 1
    $\begingroup$ As a token of gratitute, I should mention that the Otto et Villani reference you mentioned in your answer got me deep-diving into the subject of measure concentration, with a eye towards machine-learning applications. Here is my first produce arxiv.org/pdf/1810.04065.pdf $\endgroup$
    – dohmatob
    Commented Oct 19, 2018 at 10:10
2
$\begingroup$

A natural field here is Wasserstein information geometry.

See Wuchen Li, Guido Montufar: Natural gradient via optimal transport

https://arxiv.org/abs/1803.07033

For related applications see my talk

https://speakerdeck.com/lwc2017/learning-via-wasserstein-information-geometry

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ 1) You should have specified that you link to your own works. 2) What is the OP supposed to do with the links? What if the second one dies? How do they answer the question? $\endgroup$
    – Alex M.
    Commented Apr 22, 2019 at 6:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .