Let $P$ and $Q$ be two probability distributions on $\mathbb{R}$. The goal is to obtain a notion of ``distance'' between $P$ and $Q$, e.g., total variation distance, K-L divergence.

Let $f\colon \mathbb{R} \rightarrow \mathbb{R}$ be a test function. If $f$ satisfies $$ \mathbb{E}_P [ f] = \mathbb{E}_Q[ f],$$ that is, $f$ has the same expectation under $P$ and $Q$. Moreover, if $$ \text{Var}_P[f] = 0~~\text{and}~~ \text{Var}_Q [f] >0,$$ can we say that $TV(P, Q) =1$?

For example, let $P $ be the Rademacher distribution and $Q $ be the standard normal distribution. Consider $f(x) = x^2$. Then $$ \mathbb{E}_P [ f] = \mathbb{E}_Q[ f] =1, ~~\text{Var}_P[f] =0, ~~\text{Var}_Q[f] =1. $$ In this case $TV(P,Q) =1$.

As an extension, if we have $$ \text{Var}_P[f]\leq \epsilon \text{and}~~\text{Var}_Q[f] \geq C $$ for some small number $\epsilon$ and a constant $C$, can we say something about $TV(P,Q)$?


Let $f(x) = x$, $P = \delta_0$, and $Q = (1-2\epsilon)\delta_0 + \epsilon \delta_c + \epsilon \delta_{-c}$. Then $\mathbb{E}_P[f] = \mathbb{E}_Q[f] = 0$ and $\operatorname{Var}_P(f) = 0$. By taking $\epsilon$ small, you can make $TV(P,Q) = 2\epsilon$ as small as desired. By taking $c$ large, you can make $\operatorname{Var}_Q(f) = 2 \epsilon c^2$ as large as desired.

| cite | improve this answer | |
  • $\begingroup$ Thanks for your answer! If my understand is correct, do you mean when $\text{Var}_P[f] $ is extremely small and $\text{Var}_Q[f] $ is bounded below by a constant, $TV(P, Q)$ can be either large or small? $\endgroup$ – Steve Jan 21 '17 at 22:07
  • $\begingroup$ @Steve: Yes, correct. You already have an example where it is large. $\endgroup$ – Nate Eldredge Jan 21 '17 at 22:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.