Suppose we are given a class of probability density functions $\mathcal{F}$ so that for every $f \in \mathcal{F}$ we have $\alpha \leq f \leq \beta$ for some positive $\alpha, \beta \in \mathbb{R}_+$ and the density have the same support set $B$.

Here it is proved that in this setting the $L_2$ norm between density functions, i.e., $$ \|f-g\|^2_2 = \int_B (f(x) - g(x))^2 \mu(dx) $$ is equivalent to the KL divergence between $f$ and $g$, i.e., for $$ D_{\operatorname{KL}}(f \| g) = \int_B f(x) \log \frac{f(x)}{g(x)} \mu(dx) $$ we have $$ D_{\operatorname{KL}}(f \| g) \asymp \|f-g\|^2_2 $$ where $\asymp$ denotes equiality up to constant factors (which may depend on $\alpha, \beta$ but we consider those fixed).

Here a question was asked regarding the comparison of $L_1$ and $L_2$, and was argued that for such $\mathcal{F}$ classes of functions, it is possible that $$ \|f-g\|_1 = \int_B |f(x) - g(x)| \mu(dx) \ll \|f-g\|_2 $$ in general.

By Pinsker's inequality we know that $$ \|f-g\|_1 \lesssim \sqrt{D_{\operatorname{KL}}(f \| g)} \asymp \|f-g\|_2 $$ where $\lesssim$ indicates inequality up to absolute constant factors.

My question is as follows. Suppose we impose smoothness assumptions on $\mathcal{F}$. For instance suppose $\mathcal{F}$ consists of Lipschitz continuous functions with Lipschitz constant $L$. Does a reverse Pinsker's inequality hold in this case i.e. is it possible that $$ \|f-g\|_1 \gtrsim \sqrt{D_{\operatorname{KL}}(f \| g)} \asymp \|f-g\|_2\;? $$

The counterexample given here fails to work in the Lipschtiz $\mathcal{F}$ case, and is not clear how it can be recreated because the functions need to be smooth.

  • $\begingroup$ I've generally seen reverse pinsker inequalities only under assumptions that $c \leq f(x)/g(x)\leq c'$ are bounded for all $x$, see for example this. One can get something sort-of like a reverse pinsker inequality via bounds $D_{\mathsf{KL}(T(f)||T(g))} \leq C_T \lVert f -g\rVert_1^2$, where $T(f)$ is the markov kernel $T$ acting on the random variable with density $f$, and $C_T$ is a constant that depends on $T$, but not $f,g$. See for example this. Not relevant to your exact question, but hopefully useful. $\endgroup$ Mar 25 at 22:13

1 Answer 1


The answer is no.

E.g., suppose that both $f$ and $g$ are supported on $B=[0,1]$, with $f=1$ on $B$ and $$g(x)=(1-h+x)\,1(0\le x<h)+1(h\le x<1-h)+(x+h)\,1(1-h\le x\le1)$$ for $h\in(0,1/2)$ and $x\in B$.

Then $f$ and $g$ are $1$-Lipschitz on $B$, $\|f-g\|_1=h^2$, and $$D_{\operatorname{KL}}(f\|g) =-\int_0^h dx\,\ln(1-h+x)-\int_{1-h}^1 dx\,\ln(x+h) \\ =-\int_0^h dx\,\ln(1-(h-x)^2) \ge\int_0^h dx\,(h-x)^2=h^3/3,$$ so that $$\|f-g\|_1\not\gtrsim\sqrt{D_{\operatorname{KL}}(f \| g)}$$ if $h$ is small enough.

  • $\begingroup$ Awesome counterexample thanks! Do you have any thoughts on whether the exponents of $h$ can be made even more discrepant? In this example for instance $\|f-g\|_1 = h^2$ while $\|f-g\|_2 = h$. $\endgroup$ Mar 25 at 20:32
  • $\begingroup$ @spacetimewarp : Thank you for your appreciation. My guess/feeling is this is the largest discrepancy one can get given the Lipschitz condition, but I don't have a proof of this at this point. $\endgroup$ Mar 25 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.