Skip to main content
Post Undeleted by Steve
added 103 characters in body
Source Link
Steve
  • 1.1k
  • 6
  • 16

TheJust a partial answer, but the proposed inequality doesn't hold.

Take $p = [1/4, 3/4], q = [3/8, 5/8]$$p = [0.2, 0.8], q = [0.1, 0.9]$.

Then: $H(p) \approx -0.56, H(q) \approx -0.66, KL(p, q) \approx 0.04$$H(p) = 0.2 \log(5) + 0.8 \log(1/0.8) \approx 0.5$,

$H(q) = 0.1 \log(10) + 0.9 \log(1/0.9) \approx 0.33$

and hence $H(p) > H(q) + KL(p, q)$$KL(p, q) = 0.2 \log(2) + 0.8 \log(0.8/0.9) \approx 0.04$.

The proposed inequality doesn't hold.

Take $p = [1/4, 3/4], q = [3/8, 5/8]$.

Then: $H(p) \approx -0.56, H(q) \approx -0.66, KL(p, q) \approx 0.04$ and hence $H(p) > H(q) + KL(p, q)$.

Just a partial answer, but the proposed inequality doesn't hold.

Take $p = [0.2, 0.8], q = [0.1, 0.9]$.

Then $H(p) = 0.2 \log(5) + 0.8 \log(1/0.8) \approx 0.5$,

$H(q) = 0.1 \log(10) + 0.9 \log(1/0.9) \approx 0.33$

and $KL(p, q) = 0.2 \log(2) + 0.8 \log(0.8/0.9) \approx 0.04$.

Post Deleted by Steve
Source Link
Steve
  • 1.1k
  • 6
  • 16

The proposed inequality doesn't hold.

Take $p = [1/4, 3/4], q = [3/8, 5/8]$.

Then: $H(p) \approx -0.56, H(q) \approx -0.66, KL(p, q) \approx 0.04$ and hence $H(p) > H(q) + KL(p, q)$.