Skip to main content
1 of 2
MERTON
  • 505
  • 3
  • 19

The quantity $H(P)+D(P\,\|\, Q)$ is known as the cross entropy of $(P,Q)$, sometimes denoted by $H_c(P,Q)$. It holds that

$$ H_c(P,Q)\geq H(P)\;\text{ and } H_c(P,Q)\geq H_{\infty}(Q), $$

where $H_\infty$ is the min-entropy. The second inequality above is tight, by the example given above by cardinal. (The distribution $P$ deterministically chooses the maximum coordinate of $Q$)

MERTON
  • 505
  • 3
  • 19