Skip to main content
MathOverflow

Return to Answer

added 36 characters in body
Source Link
Deane Yang
Deane Yang
  • 27.5k
  • 5
  • 89
  • 180

I don't have a geometric interpretation of this inequality, but I do like to think of it as a limiting case of the Holder inequality: Given $0 < \alpha < 1$,

$$ \log \sum_i p_i\left(\frac{q_i}{p_i}\right)^\alpha \le \log\left(\sum_i q_i\right)^\alpha \left(\sum_i p_i\right)^{1-\alpha} = 0. $$$$ \frac{1}{1-\alpha}\log \sum_i p_i\left(\frac{q_i}{p_i}\right)^\alpha \le \frac{1}{1-\alpha}\log\left(\sum_i q_i\right)^\alpha \left(\sum_i p_i\right)^{1-\alpha} = 0. $$

If you now take the limit $\alpha \rightarrow 1$, you get the Gibbs inequality. The inequality above can be viewed as the analogue of the Gibbs inequality but for Renyi entropy instead of Gibbs or Shannon entropy.

I don't have a geometric interpretation of this inequality, but I do like to think of it as a limiting case of the Holder inequality: Given $0 < \alpha < 1$,

$$ \log \sum_i p_i\left(\frac{q_i}{p_i}\right)^\alpha \le \log\left(\sum_i q_i\right)^\alpha \left(\sum_i p_i\right)^{1-\alpha} = 0. $$

If you now take the limit $\alpha \rightarrow 1$, you get the Gibbs inequality. The inequality above can be viewed as the analogue of the Gibbs inequality but for Renyi entropy instead of Gibbs or Shannon entropy.

I don't have a geometric interpretation of this inequality, but I do like to think of it as a limiting case of the Holder inequality: Given $0 < \alpha < 1$,

$$ \frac{1}{1-\alpha}\log \sum_i p_i\left(\frac{q_i}{p_i}\right)^\alpha \le \frac{1}{1-\alpha}\log\left(\sum_i q_i\right)^\alpha \left(\sum_i p_i\right)^{1-\alpha} = 0. $$

If you now take the limit $\alpha \rightarrow 1$, you get the Gibbs inequality. The inequality above can be viewed as the analogue of the Gibbs inequality but for Renyi entropy instead of Gibbs or Shannon entropy.

Source Link
Deane Yang
Deane Yang
  • 27.5k
  • 5
  • 89
  • 180

I don't have a geometric interpretation of this inequality, but I do like to think of it as a limiting case of the Holder inequality: Given $0 < \alpha < 1$,

$$ \log \sum_i p_i\left(\frac{q_i}{p_i}\right)^\alpha \le \log\left(\sum_i q_i\right)^\alpha \left(\sum_i p_i\right)^{1-\alpha} = 0. $$

If you now take the limit $\alpha \rightarrow 1$, you get the Gibbs inequality. The inequality above can be viewed as the analogue of the Gibbs inequality but for Renyi entropy instead of Gibbs or Shannon entropy.