Let $E$ be a polish space, $\mathcal{P}(E)$ the Borel probability measures on $E$ with the topology of weak convergence and $\mathcal{Q} \subset \mathcal{P}(E)$ a convex and compact set.

First, the motivation for my question:

The Kullback Leibler divergence (or relative entropy) $R : \mathcal{P}(E)\times \mathcal{P}(E) \rightarrow [0, \infty]$ is defined by $R(\nu, \mu) = \int \frac{d\nu}{d\mu} \log(\frac{d\nu}{d\mu}) d\mu$ if $\nu$ is absolutely continuous with respect to $\mu$ and $R(\nu, \mu) = \infty$, else.

Let $\theta \in \mathcal{P}(E)$ and $\theta^{\ast}_R := \arg\min_{\mu \in \mathcal{Q}} R(\mu, \theta)$ be the information projection of $\theta$ onto $\mathcal{Q}$. As a consequence of the pythagorean like theorem for relative entropy (c.f. Csiszár (1975), Theorem 2.2) it holds \begin{equation} \label{eq1} \forall \mu \in \mathcal{Q} : R(\theta^{\ast}_R, \theta) + R(\mu, \theta^{\ast}_R) \leq R(\mu, \theta) \end{equation}

My question is whether there exists a metric $d$ on $\mathcal{P}(E)$ which satisfies the same kind of result. So if $\theta^{\ast}_d := \arg\min_{\mu\in\mathcal{Q}} d(\mu, \theta)$, it should hold $$ \forall \mu \in \mathcal{Q}: d(\theta^{\ast}_d, \theta)^2 + d(\mu, \theta^{\ast}_d)^2 \leq d(\mu, \theta)^2 $$

Ideally, the metric $d$ should metrize weak convergence, but I am thankful for any approaches or pointers.

  • 1
    $\begingroup$ Just as a note, the KL divergence doesn't induce the weak topology (and there are actually two separate topologies depending on whether you take open balls with respect to the first or second index). Similarly, $f$- divergences will also not induce the weak topology. $\endgroup$
    – Gabe K
    Dec 19, 2019 at 19:38


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.