Lower semicontinuity of Kullback-Leibler divergence - MathOverflow most recent 30 from http://mathoverflow.net 2013-05-21T10:35:40Z http://mathoverflow.net/feeds/question/84531 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://mathoverflow.net/questions/84531/lower-semicontinuity-of-kullback-leibler-divergence Lower semicontinuity of Kullback-Leibler divergence Michele 2011-12-29T17:37:49Z 2011-12-30T12:43:45Z <p>The Kullback-Leibler Divergence (KLD) of two PMF's $P(x)$ and $Q(x)$ is $D(P||Q)=\sum_x P(x)\log(P(x)/Q(x))$, with the provisos that $0\cdot \log (0/p)=0$ and $p\cdot \log (p/0)=+\infty$ whenever $p>0$.</p> <p>It is known that KLD is continuous at $(P,Q)$ if $Q$ is <em>strictly positive over all $x$'s</em>. What can be said otherwise?</p> <p>To be more specific, assume we are given a sequence of PMF ${(P_n,Q_n)}_{n\geq 0}$ s.t. $(P_n,Q_n)\rightarrow (P,Q)$ in the simplex of PFM's (with the topology induced by, say, norm-1 distance).</p> <p>Is it correct to deduce that</p> <p>$\lim \inf_{n\rightarrow \infty} D(P_n||Q_n) \geq D(P||Q)$ ?</p> <p>This would follows if KLD is lower-semicontinuous, right?</p> <p>Many thanks.</p> http://mathoverflow.net/questions/84531/lower-semicontinuity-of-kullback-leibler-divergence/84545#84545 Answer by Igor Rivin for Lower semicontinuity of Kullback-Leibler divergence Igor Rivin 2011-12-29T19:50:05Z 2011-12-29T19:50:05Z <p>I believe that the lower semicontinuity of KBD is proved in Cover-Thomas Information Theory book. Also in Kullback's information theory book.</p> http://mathoverflow.net/questions/84531/lower-semicontinuity-of-kullback-leibler-divergence/84582#84582 Answer by Ashok for Lower semicontinuity of Kullback-Leibler divergence Ashok 2011-12-30T08:40:17Z 2011-12-30T08:40:17Z <p>In addition to the conventions you have mentioned, it is also assumed that $0\log(0/0)=0$.</p> <p>With these conventions, I think, in the finite case, it is always true that $$\lim_{n\to \infty} D(P_n||Q_n)=D(P||Q)$$ As you said, if $Q(x)>0$ for all $x$, its immediate from the Dominated Convergence theorem. The problem is only when for some $y$, $Q(y)=0$ whereas $P(y)>0$.</p> <p>In which case $P(y)\log(P(y)/Q(y))=\infty$ and $D(P\|Q)=\infty$</p> <p>But since $(P_n,Q_n)\to (P,Q)$, we have $P_n(y)\to P(y)$ and $Q_n(y)\to Q(y)$, whence $$P_n(y)\log(P_n(y)/Q_n(y))\to P(y)\log(P(y)/Q(y))=\infty.$$ Hence $D(P_n||Q_n)\to \infty$</p> <p>So in any case we have $\lim_{n\to \infty} D(P_n||Q_n)=D(P||Q)$.</p> <p>In a general measurable space (i.e., if $P_n, Q_n, P, Q$ are probability measures on some general measure space say $(\mathbb{X}, \mathcal{X})$), I think, we have only lower semicontinuity.</p> <p>Pardon me, if something is wrong.</p>