The proposed inequality doesn't hold.
Take $p = [1/4, 3/4], q = [3/8, 5/8]$.
Then: $H(p) \approx -0.56, H(q) \approx -0.66, KL(p, q) \approx 0.04$ and hence $H(p) > H(q) + KL(p, q)$.
Steve
- 1.1k
- 6
- 16