## Bounding Entropy in terms of KL-Divergence

Let $h(X)$ be the differential entropy of a continuous random variable $X$ with density $f$, and let $Y$ be another continuous random variable with density $g$. If $KL(X\mid\mid Y)$ is the Kullback-Leibler Divergence between the two, and I know that $KL(X\mid\mid Y)<\alpha$, then can I say anything about $d(H(X),H(Y))$ where $d$ is any metric (e.g. euclidean, L1, etc.)?

I know that $KL(X\mid\mid Y) = H(X,Y)-H(X)$ where $H(X,Y)=\int f(x) \log g(x)dx$ is the cross-entropy. However, I haven't been able to do anything with this. I feel like I'm missing something pretty basic here, but haven't been able to make any progress, nor did I immediately find something in "Elements of Information Theory" (Cover and Thomas).

-
 There are some issues with your notation and definition. $H(X,Y)$ usually denoted the joint entropy of $X$ and $Y$ and $KL(X\|Y)=\int f\log (f/g)=\int f\log f-\int f\log g$ when the integrals exist. With the given information one can say that $$|H(X)-H(Y)|<\alpha+\int f\log g+H(Y) \text{ when } H(X)\le H(Y)$$ and $$|H(X)-H(Y)|>-\alpha-\int f\log g-H(Y) \text{ when } H(X)>H(Y)$$ – Ashok May 19 2012 at 6:35 Thanks for the bounds! What notation would you use for cross-entropy? As far as I can tell, there isn't an established standard. I know that I'm overloading $H(X,Y)$, but this is what I've seen elsewhere. – Ben Charrow May 21 2012 at 15:53