Suppose we have two Gibbs measures with densities $$ p_f(x) \propto \exp(f(x)),\quad q_g(x)\propto \exp(g(x)). $$ Consider the KL-divergence between $p_f$ and $q_g$, as a functional of $f$ and $g$, that is, $$ D(f, g) := \text{KL}(p_f \| q_g). $$

Question: Do we have the following lower bound: $$ D(f, g) \geq \|f - g\|^2, $$ where we are interested in, for example, the $L_2$-norm of $f-g$.

| cite | improve this question | | | | |

The answer is no. Indeed, your question can be restated as follows: Is it true that for some constant $c>0$ we have $$\int P\ln\frac PQ\,d\mu\ge c\int(\ln P-\ln Q)^2\,d\mu, \tag{1} $$ where $P$ and $Q$ are probability densities with respect to a measure $\mu$?

Let $\mu$ be the counting measure on the set $\{0,1\}$, and let $P(0)=p\in(0,1)$ and $Q(0)=q\in(0,1)$. Then (1) will become $$p\ln\frac pq+(1-p)\ln\frac{1-p}{1-q}\ge c\Big[\Big(\ln\frac pq\Big)^2+\Big(\ln\frac{1-p}{1-q}\Big)^2\Big]. \tag{2} $$ Letting now $p\to0$, we get a contradiction, because the left-hand side of (2) will go to $\ln\frac1{1-q}<\infty$, but the right-hand side of (2) will go to $\infty$.

| cite | improve this answer | | | | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.