Lower bound for KL divergence of bounded densities and $L_{2}$ metric

Question

I am currently reading "Smoothing of Multivariate Data" by Klemela. It contains Lemma 11.6, which upper and lower bounds the KL-divergence of two densities in terms of the $L_{2}$-metric. The Lemma specifically states the following 2 bounds without proof:

Upper bound

Let $f, f_{0}$ be densities. In particular, if $\inf _{x \in \mathbf{R}^{d}} f_{0}(x)>0$, then $$ D_{K}^{2}\left(f, f_{0}\right) \leq \frac{\left\|f-f_{0}\right\|_{2}^{2}}{\inf _{x \in \mathbf{R}^{d} f_{0}(x)}} $$

Lower bound

Also, if $f$ and $f_{0}$ are both bounded and bounded away from zero, then $$ D_{K}^{2}\left(f, f_{0}\right) \geq \int_{\left\{x \in \mathbf{R}^{d}: f_{0}(x)>0\right\}}\left(f-f_{0}\right)+C\left\|f-f_{0}\right\|_{2}^{2} $$ for a positive constant $C$.

I can prove the upper bound rigorously as follows: \begin{align} D_{K}^{2}(f, f_{0}) &\leq \chi^{2}(f || f_{0}) \tag{well known upper bound} \\ &=: \int_{\left\{x \in \mathbf{R}^{d}: f_{0}(x)>0\right\}} \frac{\left(f-f_{0}\right)^{2}}{f_{0}} \tag{by definition} \\ &\leq \frac{1}{\inf _{x \in \mathbf{R}^{d} f_{0}(x)}} \int_{\left\{x \in \mathbf{R}^{d}: f_{0}(x)>0\right\}} \left(f-f_{0}\right)^{2} \tag{since $\inf$ exists.} \\ &=: \frac{\left\|f-f_{0}\right\|_{2}^{2}}{\inf _{x \in \mathbf{R}^{d} f_{0}(x)}} \end{align}

Note here that the definition of the KL divergence used in the Lemma is as follows:

If $f$ and $g$ are densities of $P$ and $Q$ with respect to the Lebesgue measure, then we may write $$ D_{K}^{2}(f, g)=\int_{\mathbf{R}^{d} \cap\{x: g(x)>0\}} f \log _{e}\left(\frac{f}{g}\right) $$

I'm unsure on how to prove the lower bound. Some comments (for the lower bound to hold):

• It appears that the densities here must also be bounded above (else see first counterexample here).
• Since the densities are assumed to be bounded above and positively bounded from below over their common support, the support set in $\mathbb{R}^{d}$ must also be bounded (else see second counterexample here).

Could anyone please show how to prove this lower bound rigorously (or provide a citable rigorous proof reference)? Also, if I've made a mistake in my two comments above, please also let me know.

Aside: I had originally posted this on math.SE. To respect math.overflow cross-posting etiquette, I've deleted that post due to no responses. I realized that it had also been asked there before without any suitable responses. Since it is a research related question with no well citable proof, I believe it is fair to post here on math.overflow to settle the issue.

Answer 1

$\newcommand{\ep}{\varepsilon}$As in your post and comments, suppose that $f$ and $f_0$ are supported on a compact set $S$, and \begin{equation*} a\le f\le b,\quad a\le f_0\le b \end{equation*} on $S$ for some real $a,b$ such that $0<a<b$.

Then \begin{equation*} \int_{\left\{x \in \mathbf{R}^{d}: f_{0}(x)>0\right\}}\left(f-f_{0}\right) =\int_S\left(f-f_{0}\right)=0. \end{equation*} So, the inequality in question is simply is \begin{equation*} D_{K}^{2}\left(f, f_{0}\right) \ge C\left\|f-f_{0}\right\|_{2}^{2}. \tag{1}\label{1} \end{equation*} By definition, \begin{equation*} D_K^2(f,f_0)=\int_S f\ln\frac f{f_0}=-\int_S f\ln\frac{f_0}f. \end{equation*} We have the elementary inequality \begin{equation*} \ln x\le(x-1)-\ep_M(x-1)^2 \tag{2}\label{2} \end{equation*} for any real $M>1$ and all $x\in(0,M]$, where \begin{equation*} \ep_M:=\frac{M-1-\ln M}{(M-1)^2}>0. \end{equation*} Note that $0<\frac{f_0}f\le\frac ba$ on $S$. So, using \eqref{2} with $x=\frac{f_0}f$ and $M=\frac ba$, we get \begin{equation*} \begin{aligned} D_K^2(f,f_0)&\ge-\int_S f\Big(\frac{f_0}f-1-\ep_{b/a}\Big(\frac{f_0}f-1\Big)^2\Big) \\ & =\ep_{b/a}\int_S f\Big(\frac{f_0}f-1\Big)^2 =\ep_{b/a}\int_S \frac{(f_0-f)^2}f \\ &\ge\frac{\ep_{b/a}}b\,\int_S (f_0-f)^2 =C_{a,b}\|f-f_0\|_2^2, \end{aligned} \end{equation*} where \begin{equation*} C_{a,b}:=\frac{\ep_{b/a}}b>0, \end{equation*} as desired.