9
$\begingroup$

For a given discrete probability distribution, Shannon entropy can be though as an expectation value $\langle - \log p \rangle$ (see also: What is entropy, really?, What is the role of the logarithm in Shannon's entropy? - Stats.SE).

Are higher moments of information $\langle (- \log p)^k \rangle$ ever used?

(I can imagine some scenarios where we are not interested in the average information, but its fluctuations or 'pessimistic'/'optimistic' scenarios. However, I don't remember seeing it in applied anywhere.)

Side note: they do carry the same, ehkm, information as Rényi entropy, as they are related by $$ 2^{z H_{1-z}(X)} = \sum_i p_i^{1-z} = \sum_i p_i 2^{-z \log p_i} = \sum_k \tfrac{z^k}{k!} \sum_i p_i (- \log p_i)^k. $$

EDIT: By "higher" I mean $k>1$. So, for example, is the variance of information used for something?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ what's 'ehkm'?? $\endgroup$
    – Memming
    Commented Dec 18, 2014 at 20:59
  • $\begingroup$ @Memming An utterance to express that here it is awkward to use information in another, informal meaning. $\endgroup$
    – Piotr Migdal
    Commented Dec 18, 2014 at 22:21

1 Answer 1

Reset to default
5
$\begingroup$

In

  • H. Jürgensen, D. E. Matthews, "Entropy and Higher Moments of Information", Journal of Universal Computer Science vol 16, nr. 5 (2010)

which is available here, the authors introduce the higher moments of information of memoryless sources (section 3) and general, resp. Markov sources in sections 5 resp. 6. They move from the Shannon entropy formalism to more general entropies in section 7. The main motivation behind the introduction of higher moments is cryptographic security, as stated at pag. 785 loc. cit in the "Final Observations" section.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .