# Divergence between two random variables

I have two Gaussian random variables $X$ and $Y$, each of which is an estimator of an underlying quantity. I need to measure whether $Y$ is estimating something different than $X$. So if the mean of $Y$ is very different to the mean of $X$, that is good. However, if the variance of $Y$ is also much higher than the variance of $X$ it might just be that $Y$ is a poor estimator and the different mean is not meaningful (if you'll excuse the pun).

So for a fixed $X$, I need a function of $Y$ that increases with $\left|\mu_Y-\mu_X\right|$ but that decreases with $\sigma_Y$ (and is zero if and only if $\sigma_Y = \sigma_X$ and $\mu_Y = \mu_X$). I don't really know whether to call this a measure, a divergence or something else, since it will be negative when $\mu_Y = \mu_X$ and $\sigma_Y > \sigma_X$.

I thought that the Kullback-Leibler divergence might be the way to go but it responds the wrong way to changes in the variance of $Y$. Also, it can't be negative.

Clearly, something like $\left|\mu_Y-\mu_X\right| + \sigma_X - \sigma_Y$ has the above properties but I'm looking for something a little more grounded in information theory.

Ideally, this would generalize to distributions other than Gaussian.

-
Could you explain why KL-divergence is not suitable in a bit more detail? –  VSJ Mar 9 '12 at 3:30
Your "if and only if" statement can't work if your "divergence" $f$ is a continuous function: every curve from a point where $f > 0$ to a point where $f < 0$ must intersect the set where $f = 0$. –  Robert Israel Mar 9 '12 at 4:38
VSJ: KL-divergence would think that a $Y$ with a large variance was very different to $X$. I want something that penalizes variance. Robert: Yes, of course. I should have said something "is zero if $X = Y$". That is all I really wanted and I do want $f$ to be continuous. –  user21997 Mar 12 '12 at 23:37

Let $0<\alpha<1$ (typically $\alpha=0.05$) and choose $\varepsilon>0$ such that

$$\alpha = \mathbb P(\left |X-\mu_X\right | > \varepsilon)$$

Now let

$$f = \mathbb P(\left |Y-\mu_X\right | > \varepsilon) - \alpha$$

Then

1. $f$ is increasing with $|\mu_X-\mu_Y|$,

2. $f$ is decreasing with $\sigma_Y$, and

3. $f=0$ when $\mu_X=\mu_Y$ and $\sigma_X=\sigma_Y$.

This approach comes not so much from information theory as from hypothesis testing, in particular the notion of statistical power. Some generalizing from the case of the normal distribution seems possible, but note that in (3) we need the distribution to be determined by $\mu_Y$ and $\sigma_Y$.

-
Many thanks Bjørn. I'll definitely give this a try. I've just started reading "Information Theory, Inference and Learning Algorithms" by MacKey which promises to connect information theory and statistical inference. Perhaps I'll have a better understanding of this once I'm done with that. –  user21997 Mar 12 '12 at 23:40