Bounding Entropy in terms of KL-Divergence - MathOverflow most recent 30 from http://mathoverflow.net 2013-05-19T22:40:52Z http://mathoverflow.net/feeds/question/97324 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://mathoverflow.net/questions/97324/bounding-entropy-in-terms-of-kl-divergence Bounding Entropy in terms of KL-Divergence Ben Charrow 2012-05-18T16:48:32Z 2012-05-18T16:48:32Z <p>Let $h(X)$ be the differential entropy of a continuous random variable $X$ with density $f$, and let $Y$ be another continuous random variable with density $g$. If $KL(X\mid\mid Y)$ is the Kullback-Leibler Divergence between the two, and I know that $KL(X\mid\mid Y)&lt;\alpha$, then can I say anything about $d(H(X),H(Y))$ where $d$ is any metric (e.g. euclidean, L1, etc.)?</p> <p>I know that $KL(X\mid\mid Y) = H(X,Y)-H(X)$ where $H(X,Y)=\int f(x) \log g(x)dx$ is the cross-entropy. However, I haven't been able to do anything with this. I feel like I'm missing something pretty basic here, but haven't been able to make any progress, nor did I immediately find something in "Elements of Information Theory" (Cover and Thomas).</p>