Skip to main content
2 of 2
added 103 characters in body
Steve
  • 1.1k
  • 6
  • 16

Just a partial answer, but the proposed inequality doesn't hold.

Take $p = [0.2, 0.8], q = [0.1, 0.9]$.

Then $H(p) = 0.2 \log(5) + 0.8 \log(1/0.8) \approx 0.5$,

$H(q) = 0.1 \log(10) + 0.9 \log(1/0.9) \approx 0.33$

and $KL(p, q) = 0.2 \log(2) + 0.8 \log(0.8/0.9) \approx 0.04$.

Steve
  • 1.1k
  • 6
  • 16