# Distance of distributions of random variables, without PDF

Consider an interval $I$ with a smooth probability measure $d\mu (x) = c(x) dx$ and two known real measurable functions $f_1(x)$,$f_2(x)$. Both functions define a distribution on $X = {\rm Im} \, [f_1] \cup {\rm Im} \, [f_2],$ the distributions are denoted $\rho_{1,2}$, respectively.

My questions:

1. Is there a sense of distance between $\rho _1$ and $\rho _2$ that does not involve their explicit computation, but only of the functions $f$?
2. Does that answer changes if both $f$ are smooth?
3. If both their images are bounded?
4. If $f_1(x) = x$, and $d\mu = dx$, and so $\rho _1$ is the uniform distribution?

My motivation: Numerical computation of $\rho$ is notoriously inaccurate and problematic, and so KL divergence or even $L^1$ distance are problematic as well. I obtain $f$ anyway, and I really only need to know how "far" is $\rho$ from being uniform.

Disclaimer: Having a "good distance measure" is vague, and can be interpreted in many ways. I know. But the ways to define it I already know involve the computation of $\rho$, so I'd rather leave it open-ended for now.

• Are you willing to add the assumption that the two functions are increasing? In that case, the integral of the difference is a measure of the distance between the distributions. It's also reasonably robust. Feb 27 '17 at 4:19
• Thanks! Unfortunately no, they're not monotone. @AnthonyQuas Feb 27 '17 at 17:44
• There is always an increasing rearrangement. Feb 27 '17 at 18:30
• @AnthonyQuas Even if they're not smooth? How? Feb 27 '17 at 21:26
• In the case where $\mu$ is Lebesgue, you just take the (compositional) inverse of the cdf. Feb 27 '17 at 23:22

Numerical computation of $\rho$ is notoriously inaccurate and problematic, and so KL divergence or even $L^1$ distance are problematic as well. I obtain $f$ anyway, and I really only need to know how "far" is $\rho$ from being uniform.

And I assumed that your already obtained some information about the distribution of $f_i$, say moments. The moments of $f_i$'s are most easily obtained if you know the original $d\mu$ via simulation techniques like MCMC. Then I would suggest Wasserstein distance as a candidate since all you want is how dissimilar it looks from uniform, and Wasserstein distance is proven to perform well in this respect.

Your calculation does not have to involve $\rho_i$ but you should at least be able to sample from $d\mu$ and hence $f_i(X)$.

And the following paper verified that the convergence behavior is satisfying.

Fournier, Nicolas, and Arnaud Guillin. "On the rate of convergence in Wasserstein distance of the empirical measure." Probability Theory and Related Fields 162.3-4 (2015): 707-738.

• Thanks! It doesn't seem from the paper/the Wiki page that the Wasserstein distance is easily computable. Am I wrong? Mar 3 '17 at 9:19
• @AmirSagiv I think Wasserstein distance is rather easy to compute since you only have to know $\mu$ and know the moments from $f_i$. I do not think there could be an easier one unless I know what you actually know about $f_i$ further. Mar 3 '17 at 20:02
• Don't I need to take an infimum over all measures? Mar 3 '17 at 21:08
• Yes but you can do it using simple variantional technique, since you have only finite samples this should not cost too much computational time. Mar 3 '17 at 21:19