Skip to main content
1 of 2
Deane Yang
  • 27.5k
  • 5
  • 89
  • 180

I don't have a geometric interpretation of this inequality, but I do like to think of it as a limiting case of the Holder inequality: Given $0 < \alpha < 1$,

$$ \log \sum_i p_i\left(\frac{q_i}{p_i}\right)^\alpha \le \log\left(\sum_i q_i\right)^\alpha \left(\sum_i p_i\right)^{1-\alpha} = 0. $$

If you now take the limit $\alpha \rightarrow 1$, you get the Gibbs inequality. The inequality above can be viewed as the analogue of the Gibbs inequality but for Renyi entropy instead of Gibbs or Shannon entropy.

Deane Yang
  • 27.5k
  • 5
  • 89
  • 180