# Distances between probability distributions by the variance of the test functions

Let $P$ and $Q$ be two probability distributions on $\mathbb{R}$. The goal is to obtain a notion of distance'' between $P$ and $Q$, e.g., total variation distance, K-L divergence.

Let $f\colon \mathbb{R} \rightarrow \mathbb{R}$ be a test function. If $f$ satisfies $$\mathbb{E}_P [ f] = \mathbb{E}_Q[ f],$$ that is, $f$ has the same expectation under $P$ and $Q$. Moreover, if $$\text{Var}_P[f] = 0~~\text{and}~~ \text{Var}_Q [f] >0,$$ can we say that $TV(P, Q) =1$?

For example, let $P$ be the Rademacher distribution and $Q$ be the standard normal distribution. Consider $f(x) = x^2$. Then $$\mathbb{E}_P [ f] = \mathbb{E}_Q[ f] =1, ~~\text{Var}_P[f] =0, ~~\text{Var}_Q[f] =1.$$ In this case $TV(P,Q) =1$.

As an extension, if we have $$\text{Var}_P[f]\leq \epsilon \text{and}~~\text{Var}_Q[f] \geq C$$ for some small number $\epsilon$ and a constant $C$, can we say something about $TV(P,Q)$?

## 1 Answer

Let $f(x) = x$, $P = \delta_0$, and $Q = (1-2\epsilon)\delta_0 + \epsilon \delta_c + \epsilon \delta_{-c}$. Then $\mathbb{E}_P[f] = \mathbb{E}_Q[f] = 0$ and $\operatorname{Var}_P(f) = 0$. By taking $\epsilon$ small, you can make $TV(P,Q) = 2\epsilon$ as small as desired. By taking $c$ large, you can make $\operatorname{Var}_Q(f) = 2 \epsilon c^2$ as large as desired.

• Thanks for your answer! If my understand is correct, do you mean when $\text{Var}_P[f]$ is extremely small and $\text{Var}_Q[f]$ is bounded below by a constant, $TV(P, Q)$ can be either large or small? – Steve Jan 21 '17 at 22:07
• @Steve: Yes, correct. You already have an example where it is large. – Nate Eldredge Jan 21 '17 at 22:18