8
$\begingroup$

Let $\mu$ and $\nu$ be two multivariate Gaussian measures on $\mathbb{R}^d$ with non-singular covariance matrices. Can the Fisher-Rao distance $d(\mu,\nu)$ computed on the information manifold of non-generated $d$-dimensional Gaussian measures with Fisher-Rao metric, be bounded by the (symmetrized) KL divergence/relative-entropy between $\mu$ and $\nu$?

$\endgroup$
3
  • $\begingroup$ What do you mean by " bounded by"? $\endgroup$ Dec 25, 2022 at 0:50
  • $\begingroup$ Bounded above by the KL divergence. $\endgroup$ Dec 25, 2022 at 16:09
  • 1
    $\begingroup$ That is, your question is whether $d(\mu,\nu)\le KL(\mu,\nu)$, where $KL(\mu,\nu)$ is the KL divergence between $\mu$ and $\nu$? $\endgroup$ Dec 25, 2022 at 16:24

1 Answer 1

7
$\begingroup$

Since relative entropy behaves locally like a squared distance, we might expect the squared Fisher-Rao metric to be comparable to the symmetrized KL divergence. This is indeed the case.

Let $d_F$ denote the Fisher-Rao metric on the manifold of non-degenerate multivariate Gaussians, and let $D(\mu,\nu):= D_{KL}(\mu\|\nu) + D_{KL}(\nu\|\mu)$ denote the symmetrized KL divergence between measures $\mu,\nu$.

Claim: For multivariate Gaussian measures $\mu_1,\mu_2$ with nonsingular covariance matrices, we have $$ d_F(\mu_1,\mu_2)^2 \leq 2 D(\mu_1,\mu_2). $$

Proof: By the triangle inequality, we have \begin{align*} d_F\big(N(\theta_1,\Sigma_1),N(\theta_2,\Sigma_2)\big)^2 &\leq \big(d_F(N(\theta_1,\Sigma_1),N(\theta_1,\Sigma_2) )+d_F(N(\theta_1,\Sigma_2),N(\theta_2,\Sigma_2) ) \big)^2\\ &\leq 2 d_F\big(N(\theta_1,\Sigma_1),N(\theta_1,\Sigma_2)\big)^2 + 2 d_F\big(N(\theta_1,\Sigma_2),N(\theta_2,\Sigma_2)\big)^2 \end{align*}

On the submanifold of Gaussians with common mean, the (squared) Fisher-Rao distance is equal to $$ d_F\big(N(\theta_1,\Sigma_1),N(\theta_1,\Sigma_2)\big)^2 = \frac{1}{2}\sum_{i}(\log \lambda_i)^2, $$
where $(\lambda_i)$ denote the eigenvalues of the matrix $\Sigma_2^{-1/2}\Sigma_1\Sigma_2^{-1/2}$. On the submanifold of Gaussians with common covariance, the (squared) Fisher-Rao distance is equal to $$ d_F\big(N(\theta_1,\Sigma_2),N(\theta_2,\Sigma_2)\big)^2 = (\theta_1-\theta_2)^T \Sigma_2^{-1} (\theta_1-\theta_2). $$

The symmetrized KL divergence is given by $$ D(N(\theta_1,\Sigma_1),N(\theta_2,\Sigma_2)) = \frac{1}{2}\sum_{i}\left(\lambda_i +\frac{1}{\lambda_i}- 2 \right) +\frac{1}{2} (\theta_1-\theta_2)^T (\Sigma_1^{-1}+\Sigma_2^{-1}) (\theta_1-\theta_2). $$

Now, the claim follows on account of the inequality $$ (\log x)^2 \leq x + \frac{1}{x}-2, ~~~x>0. $$

Remark: The closed-form expressions for the special cases of the Fisher-Rao metric used above can be found in Section 2.1 (and references therein) of:

Pinele, Julianna, João E. Strapasson, and Sueli IR Costa. "The Fisher-Rao distance between multivariate normal distributions: Special cases, bounds and applications." Entropy 22.4 (2020): 404.

$\endgroup$
2
  • $\begingroup$ Very interesting; this result seems known. Is there a standard reference? $\endgroup$ Dec 26, 2022 at 0:04
  • 1
    $\begingroup$ I don’t know of one, but I’m not very familiar with the information geometry literature. $\endgroup$
    – Tom
    Dec 26, 2022 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.