Given two probability density functions (PDF) $f$ and $g$, both defined over the same set $X$, there are many ways to describe/measure the distance between them, e.g., KL divergence and Hellinger distance, to name a few.

What I'm looking for is a distance function $d(f,g)$ that gives an order of magnitude difference. To be concrete, if at some area $f\sim 2\times 10^4$, $f_1 \sim 10^4$, and $f_2 \sim 10^2$, I want that $d(f,f_1) \ll d(f,f_2)$. Is there a known, statistically or probabilistically meaningful way, to do that?

I know that this question is a bit vague, but if I knew how to define such a distance function, I wouldn't be asking this question. Note that, as in the examples above, $d(\cdot, \cdot)$ doesn't have to be a metric.


Edit: This has been cross-posted in Cross.Validated

  • $\begingroup$ What's wrong with the $L^\infty$ norm? $\endgroup$ Sep 11 '17 at 15:30
  • $\begingroup$ Hi @SteveHuntsman, thanks for the suggestion. In my case $d(f,f_1)=10^4 \approx d(f,f_2)$. So it doesn't "respect" order of magnitude. Furthermore, it'd be prefered for the proposed distance to have a probabilistic meaning. $\endgroup$
    – Amir Sagiv
    Sep 11 '17 at 16:30
  • 3
    $\begingroup$ @AmirSagiv- Let me recommend Deza and Deza's enclyclopedia of distances at dx.doi.org/10.1007/978-3-662-44342-2_14 for a place to start $\endgroup$ Sep 11 '17 at 16:57
  • 1
    $\begingroup$ What sort of probabilistically meaningful information do you hope such a distance metric will describe? Maybe I am being cynical but I can't see how it would preserve much... you can easily design distributions that are distant in your sense [up to some interpretation] but close in many other very strong ways. $\endgroup$ Oct 3 '17 at 0:23
  • $\begingroup$ What I want to know is whether for some value $X=x_0$, I have that $f(x_0)$ is at the same order of magnitude as $f_1(x_0)$, but two order of magnitudes greater than $f_2$. So, in general, it would be some integral of this kind of a measurement, over all $X=x$. $\endgroup$
    – Amir Sagiv
    Oct 3 '17 at 12:00

What about the symmetric-ized KL divergence? $$D(p,q) + D(q,p)$$

Recalling that $$D(p,q) = \sum_x p(x) \log \left(\frac{p(x)}{q(x)}\right)$$ then the $\log(p/q)$ is the order of magnitude of the ratio.

I guess I'm assuming that you're interested in the order of magnitude of the ratio rather than the absolute value. Also, depending on what you want with the "$\ll$", you could exponentiate the distance function above. (E.g., if we consider "1000" to be 3 orders of magnitude, do you want the value "1000" or the value "3"? Symmetric KL gives you something more like "3".)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.