Given two probability density functions (PDF) $f$ and $g$, both defined over the same set $X$, there are many ways to describe/measure the distance between them, e.g., KL divergence and Hellinger distance, to name a few.

What I'm looking for is a distance function $d(f,g)$ that gives an order of magnitude difference. To be concrete, if at some area $f\sim 2\times 10^4$, $f_1 \sim 10^4$, and $f_2 \sim 10^2$, I want that $d(f,f_1) \ll d(f,f_2)$. Is there a known, statistically or probabilistically meaningful way, to do that?

I know that this question is a bit vague, but if I knew how to define such a distance function, I wouldn't be asking this question. Note that, as in the examples above, $d(\cdot, \cdot)$ doesn't have to be a metric.


Edit: This has been cross-posted in Cross.Validated

  • $\begingroup$ What's wrong with the $L^\infty$ norm? $\endgroup$ – Steve Huntsman Sep 11 '17 at 15:30
  • $\begingroup$ Hi @SteveHuntsman, thanks for the suggestion. In my case $d(f,f_1)=10^4 \approx d(f,f_2)$. So it doesn't "respect" order of magnitude. Furthermore, it'd be prefered for the proposed distance to have a probabilistic meaning. $\endgroup$ – Amir Sagiv Sep 11 '17 at 16:30
  • 1
    $\begingroup$ @AmirSagiv- Let me recommend Deza and Deza's enclyclopedia of distances at dx.doi.org/10.1007/978-3-662-44342-2_14 for a place to start $\endgroup$ – Steve Huntsman Sep 11 '17 at 16:57
  • $\begingroup$ What sort of probabilistically meaningful information do you hope such a distance metric will describe? Maybe I am being cynical but I can't see how it would preserve much... you can easily design distributions that are distant in your sense [up to some interpretation] but close in many other very strong ways. $\endgroup$ – enthdegree Oct 3 '17 at 0:23
  • $\begingroup$ What I want to know is whether for some value $X=x_0$, I have that $f(x_0)$ is at the same order of magnitude as $f_1(x_0)$, but two order of magnitudes greater than $f_2$. So, in general, it would be some integral of this kind of a measurement, over all $X=x$. $\endgroup$ – Amir Sagiv Oct 3 '17 at 12:00

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.