# log-like distance between probability distributions

Given two probability density functions (PDF) $f$ and $g$, both defined over the same set $X$, there are many ways to describe/measure the distance between them, e.g., KL divergence and Hellinger distance, to name a few.

What I'm looking for is a distance function $d(f,g)$ that gives an order of magnitude difference. To be concrete, if at some area $f\sim 2\times 10^4$, $f_1 \sim 10^4$, and $f_2 \sim 10^2$, I want that $d(f,f_1) \ll d(f,f_2)$. Is there a known, statistically or probabilistically meaningful way, to do that?

I know that this question is a bit vague, but if I knew how to define such a distance function, I wouldn't be asking this question. Note that, as in the examples above, $d(\cdot, \cdot)$ doesn't have to be a metric.

Thanks

Edit: This has been cross-posted in Cross.Validated

• What's wrong with the $L^\infty$ norm? Sep 11 '17 at 15:30
• Hi @SteveHuntsman, thanks for the suggestion. In my case $d(f,f_1)=10^4 \approx d(f,f_2)$. So it doesn't "respect" order of magnitude. Furthermore, it'd be prefered for the proposed distance to have a probabilistic meaning. Sep 11 '17 at 16:30
• @AmirSagiv- Let me recommend Deza and Deza's enclyclopedia of distances at dx.doi.org/10.1007/978-3-662-44342-2_14 for a place to start Sep 11 '17 at 16:57
• What sort of probabilistically meaningful information do you hope such a distance metric will describe? Maybe I am being cynical but I can't see how it would preserve much... you can easily design distributions that are distant in your sense [up to some interpretation] but close in many other very strong ways. Oct 3 '17 at 0:23
• What I want to know is whether for some value $X=x_0$, I have that $f(x_0)$ is at the same order of magnitude as $f_1(x_0)$, but two order of magnitudes greater than $f_2$. So, in general, it would be some integral of this kind of a measurement, over all $X=x$. Oct 3 '17 at 12:00

What about the symmetric-ized KL divergence? $$D(p,q) + D(q,p)$$
Recalling that $$D(p,q) = \sum_x p(x) \log \left(\frac{p(x)}{q(x)}\right)$$ then the $$\log(p/q)$$ is the order of magnitude of the ratio.
I guess I'm assuming that you're interested in the order of magnitude of the ratio rather than the absolute value. Also, depending on what you want with the "$$\ll$$", you could exponentiate the distance function above. (E.g., if we consider "1000" to be 3 orders of magnitude, do you want the value "1000" or the value "3"? Symmetric KL gives you something more like "3".)