For a given discrete probability distribution, Shannon entropy can be though as an expectation value $\langle - \log p \rangle$ (see also: What is entropy, really?, What is the role of the logarithm in Shannon's entropy? - Stats.SE).

Are higher moments of information $\langle (- \log p)^k \rangle$ ever used?

(I can imagine some scenarios where we are not interested in the average information, but its fluctuations or 'pessimistic'/'optimistic' scenarios. However, I don't remember seeing it in applied anywhere.)

Side note: they do carry the same, ehkm, information as Rényi entropy, as they are related by $$ 2^{z H_{1-z}(X)} = \sum_i p_i^{1-z} = \sum_i p_i 2^{-z \log p_i} = \sum_k \tfrac{z^k}{k!} \sum_i p_i (- \log p_i)^k. $$

EDIT: By "higher" I mean $k>1$. So, for example, is the variance of information used for something?

  • 2
    $\begingroup$ what's 'ehkm'?? $\endgroup$ – Memming Dec 18 '14 at 20:59
  • $\begingroup$ @Memming An utterance to express that here it is awkward to use information in another, informal meaning. $\endgroup$ – Piotr Migdal Dec 18 '14 at 22:21


  • H. Jürgensen, D. E. Matthews, "Entropy and Higher Moments of Information", Journal of Universal Computer Science vol 16, nr. 5 (2010)

which is available here, the authors introduce the higher moments of information of memoryless sources (section 3) and general, resp. Markov sources in sections 5 resp. 6. They move from the Shannon entropy formalism to more general entropies in section 7. The main motivation behind the introduction of higher moments is cryptographic security, as stated at pag. 785 loc. cit in the "Final Observations" section.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.