Take the 2-minute tour ×
MathOverflow is a question and answer site for professional mathematicians. It's 100% free, no registration required.

Let $H$ be a function of finite sequences of probabilities (non-negative numbers summing up to 1) into real numbers, such that:

  • $H$ is continuous,
  • $H$ is symmetric w.r.t. the order of its arguments,
  • $H$ is additive, i.e. $H(P \otimes Q) = H(P) + H(Q)$,

where $P = \{p_i\}_i$, $Q = \{q_i\}_i$, $P \otimes Q = \{p_i q_j\}_{i,j}$. (A physicist would call the last property extensive.)

One class of examples is Rényi entropy $H_q$, along with linear combinations of $H_q$ (for different $q$s), what was already pointed out in the original paper:

Which other, if any, functions $H$ fulfill the above postulates?

share|improve this question
if you insist on additivity, the Rényi entropy gives the most general form. –  Carlo Beenakker Mar 1 '14 at 15:35
@CarloBeenakker Is it proven somewhere? (Or obvious for some reason?) I looked at $e^H$ and it seamed to me that is need to have form $\prod_q \sum_i p_i^q$ (to make it symmetric and additive), but I don't know how to actually prove it. –  Piotr Migdal Mar 1 '14 at 15:48
What do you mean by continuous? There are several possible topologies on the space of finitely supported probability measures. –  Nate Eldredge Mar 1 '14 at 15:58
@NateEldredge "I can negotiate" (in Rényi's paper it was continuous as a function of $H(p,1-p)$). Here $L^1$ seems to be the most natural norm (unless you suggest sth different). –  Piotr Migdal Mar 1 '14 at 16:05

1 Answer 1

Rényi conjectured that his generalized entropy is the most general form that satisfies addititivity. This was proven by Daróczy,

Z. Daróczy, Über die gemeinsame Charakterisierung der zu den nicht vollständigen Verteilungen gehörigen Entropien von Shannon und von Rényi. (1963)

J. Aczel and Z. Daróczy, Charakterisierung der Entropien positiver Ordnung und der Shannonschen Entropie. (1963)

Z. Daróczy, Über Mittelwerte und Entropien vollständiger Wahrscheinlichkeitsverteilungen. (1964)

share|improve this answer
Thanks! Unfortunately the papers are paywalled for me. I can see preview for one, and it seems it uses one more axiom (related to $H(P\cup Q)$ and a generalized mean with function $g$; alike in the Rényi's paper). –  Piotr Migdal Mar 1 '14 at 16:21
@PiotrMigdal: hi again! I just came across the exact same problem. The papers that Carlo linked to indeed seem to make additional assumptions about the particular form of $H$. I can email the papers to you if you like. Have you found a satisfactory answer in the meantime? BTW one simple observation is that all linear combinations of Rényi entropies satisfy your conditions as well, so Carlo (resp Rényi) must have had something else in mind. –  Tobias Fritz Oct 27 '14 at 23:09
@TobiasFritz Hi! I would be grateful for sending them to me. I wasn't working on it in the meantime. –  Piotr Migdal Oct 28 '14 at 7:40

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.