3
$\begingroup$

I have two Gaussian random variables $X$ and $Y$, each of which is an estimator of an underlying quantity. I need to measure whether $Y$ is estimating something different than $X$. So if the mean of $Y$ is very different to the mean of $X$, that is good. However, if the variance of $Y$ is also much higher than the variance of $X$ it might just be that $Y$ is a poor estimator and the different mean is not meaningful (if you'll excuse the pun).

So for a fixed $X$, I need a function of $Y$ that increases with $\left|\mu_Y-\mu_X\right|$ but that decreases with $\sigma_Y$ (and is zero if and only if $\sigma_Y = \sigma_X$ and $\mu_Y = \mu_X$). I don't really know whether to call this a measure, a divergence or something else, since it will be negative when $\mu_Y = \mu_X$ and $\sigma_Y > \sigma_X$.

I thought that the Kullback-Leibler divergence might be the way to go but it responds the wrong way to changes in the variance of $Y$. Also, it can't be negative.

Clearly, something like $\left|\mu_Y-\mu_X\right| + \sigma_X - \sigma_Y$ has the above properties but I'm looking for something a little more grounded in information theory.

Ideally, this would generalize to distributions other than Gaussian.

$\endgroup$
3
  • $\begingroup$ Could you explain why KL-divergence is not suitable in a bit more detail? $\endgroup$
    – VSJ
    Mar 9, 2012 at 3:30
  • 2
    $\begingroup$ Your "if and only if" statement can't work if your "divergence" $f$ is a continuous function: every curve from a point where $f > 0$ to a point where $f < 0$ must intersect the set where $f = 0$. $\endgroup$ Mar 9, 2012 at 4:38
  • 1
    $\begingroup$ VSJ: KL-divergence would think that a $Y$ with a large variance was very different to $X$. I want something that penalizes variance. Robert: Yes, of course. I should have said something "is zero if $X = Y$". That is all I really wanted and I do want $f$ to be continuous. $\endgroup$
    – user21997
    Mar 12, 2012 at 23:37

1 Answer 1

3
$\begingroup$

Let $0<\alpha<1$ (typically $\alpha=0.05$) and choose $\varepsilon>0$ such that

$$\alpha = \mathbb P(\left |X-\mu_X\right | > \varepsilon)$$

Now let

$$f = \mathbb P(\left |Y-\mu_X\right | > \varepsilon) - \alpha$$

Then

  1. $f$ is increasing with $|\mu_X-\mu_Y|$,

  2. $f$ is decreasing with $\sigma_Y$, and

  3. $f=0$ when $\mu_X=\mu_Y$ and $\sigma_X=\sigma_Y$.

This approach comes not so much from information theory as from hypothesis testing, in particular the notion of statistical power. Some generalizing from the case of the normal distribution seems possible, but note that in (3) we need the distribution to be determined by $\mu_Y$ and $\sigma_Y$.

$\endgroup$
3
  • $\begingroup$ Many thanks Bjørn. I'll definitely give this a try. I've just started reading "Information Theory, Inference and Learning Algorithms" by MacKey which promises to connect information theory and statistical inference. Perhaps I'll have a better understanding of this once I'm done with that. $\endgroup$
    – user21997
    Mar 12, 2012 at 23:40
  • $\begingroup$ Is it possible to consider the entropy of $|Y-\mu_X|>\epsilon$ as some kind of divergence? I mean, it would have an interesting interpretation in terms of such an RV. $\endgroup$
    – Nacho
    Apr 6, 2021 at 1:51
  • 1
    $\begingroup$ @Nacho hmm, I'm not sure... you may consider formulating it as a Question rather than a Comment so you can add some background on what your goal is. $\endgroup$ Apr 6, 2021 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.