I am interested in a lower bound on the Bhattacharya distance between two independent multivariate Gaussian distributions. To be precise, consider zero-mean independent Gaussian distributions $p_1\sim\mathcal{N}(\mathbb{0},\Theta_1^{-1})$ and $p_2\sim\mathcal{N}(\mathbb{0},\Theta_2^{-1})$ (note that $\Theta_1$, $\Theta_2$ are $n\times n$ positive definite matrices). Then the Bhattacharya distance between $p_1$ and $p_2$ can be expressed as:

\begin{align*} d_B(p_1,p_2)=\frac{1}{2} \ln\left(\frac{|\Theta_1+\Theta_2)/2|}{|\Theta_2|^{1/2}|\Theta_2|^{1/2}}\right). \end{align*}

I would prefer the lower bound is in terms of some matrix norm of $(\Theta_1-\Theta_2)$, for e.g., $||\Theta_1-\Theta_2||_1$ (L1 norm) or $||\Theta_1-\Theta_2||_2$ (L2 norm) or $||\Theta_1-\Theta_2||_F$ (Frobenius norm).

One may note that $2d_B(p_1,p_2)=D(p_1||p)+D(p_2||p)$, where $p=(p_1+p_2)/2$ and $D(\cdot||\cdot)$ is the KL-divergence function, but I could not make use of this property to get any answer.

  • $\begingroup$ What is $|\Theta|$? And by the $L^1$ and $L^2$ norms, do you mean the induced operator norms? $\endgroup$ – Mark Meckes Sep 21 '12 at 18:42
  • $\begingroup$ $|\Theta|$ is the determinant of $\Theta$. By L1 and L2 norms, I meant matrix norms (mathworld.wolfram.com/MatrixNorm.html). $\endgroup$ – adas Sep 21 '12 at 20:11
  • 2
    $\begingroup$ Roughly speaking, if $\Theta_1^{-1}\Theta_2 \approx I$, then $d_B$ can be lower bounded by $\frac{1}{8}\|\Theta_1-\Theta_2\|^2$, but in general, there seems to be no uniform lower bound ... $\endgroup$ – Suvrit Sep 23 '12 at 11:45

The KL divergence is given by $$\hbox{Tr}(\Sigma_0\Sigma_1^{-1})- \ln(|\Sigma_0\Sigma_1^{-1}|)$$ Let $\mu_{-}$ be the smallest eigenvalue of $\Sigma_0\Sigma_1^{-1}$ and $\mu_{+}$ the largest, then

$$\mu_{-} = ||\Sigma_1 \Sigma_0^{-1} ||_2$$

$$\mu_{+} = ||\Sigma_0 \Sigma_1^{-1} ||_2$$

$$\hbox{Tr}(\Sigma_0\Sigma_1^{-1}) > n\mu_{-}$$ $$-\ln(|\Sigma_0\Sigma_1^{-1}|) > -n\ln(\mu_{+})$$ $$KL > n ||\Sigma_1\Sigma_0^{-1}||_2 -n\ln(||\Sigma_0\Sigma_1^{-1}||_2)$$

With the lower bound on the KL you can get a lower bound on the distance you're looking for. It doesn't seem you can get away with looking at the norm of the difference between the matrices.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.