Let $X$ be a random variable taking values in $\{1,\ldots,n\}$, and let $p_i$ denote the probability of the event $\{X = i\}$. Shannon defined the entropy of $X$ to be the quantity

$$H(X) = -\sum_i p_i \log p_i$$

with the convention $x \log x = 0$. This definition is the starting point for all of information theory, and consequently it has been provided with numerous axiomatic characterizations. Shannon gave a characterization in his original paper; Myron Tribus gave another involving how "surprising" a random variable is on average; yet another approach constructs entropy as the objective function for certain optimization problems (turning the principle of maximum entropy into a definition); and Baez, Fritz, and Leinster gave still another characterization involving convexity and functorial properties. I'm sure this list is not exhaustive.

But I have only ever seen axiomatic characterizations for discrete random variables. Shannon himself defined the entropy of a continuous random variable by:

$$H(X) = -\int p(x) \log p(x)\, dx$$

where $p(x)$ is the density function of $X$. Jaynes argued here that this is the wrong definition because it has the wrong units and it transforms incorrectly under a change of coordinates; he was able to modify the definition accordingly by taking a limit of discrete entropies. Regardless of which definition is correct my question is this:

Is there an axiomatic characterization of the entropy of a continuous random variable which generalizes a corresponding characterization in the discrete case? If so, what is the "right" class of measure spaces to which this characterization applies?


How about using Tribus' surprise-based characterization again? After all if $X $ and $Y $ are independent random variables with pdfs $f_X $, $f_Y $, then the joint density $f_{X,Y}(x,y) $ is $f_X(x)f_Y(y)$ and so taking log converts it into a sum, as in the discrete argument.

  • $\begingroup$ This idea (as expressed, say, here: en.wikipedia.org/wiki/Self-information) is quite compelling, but as I tried to work it out in the continuous case I realized that I had doubts even about the discrete case. Consider the fair coin flip sample space $\{H, T\}$ with P(H) = P(T) = 1/2. The event $\{H, T\}$ is independent of $H$ and $T$, as is the event $\emptyset$, but monotonicity and independence only give $S(\{H, T\}) = 0$ and $S(\emptyset) = \infty$. There are no additional pairs of independent events to fix the values of $S(H)$ and $S(T)$, so it seems any positive value will do. $\endgroup$ Feb 19 '17 at 14:55
  • $\begingroup$ Well but $S (H)=S (T) $ since it's a fair coin. And then an arbitrary positive value is of course fixed to equal 1... $\endgroup$ Feb 19 '17 at 16:59
  • $\begingroup$ Why $1$? The monotonicity condition forces $0 < S(H) = S(T) < \infty$, but there are no tools with which the additivity condition can narrow it down further. Of course $S$ is only well defined up to a constant, so my example wasn't very good; let's modify it so that $P(H) = 1/4$ and $P(T) = 3/4$. Then we expect $S(H) = \log 4$ and $S(T) = \log(4/3)$, but where do we run afoul of the axioms if we instead set $S(H) = \log 4$ and $S(T) = \log 2$ for instance? $\endgroup$ Feb 19 '17 at 17:36
  • $\begingroup$ I suppose once you have the surprise of an event of probability $1/2^n $ for each $n $ then adding the assumption that the surprise function is real analytic you can deduce the value at other points. $\endgroup$ Feb 19 '17 at 22:50
  • $\begingroup$ Obtaining events of probability $1/2^n$ already forces one to define $S$ on a new probability space whose points correspond to finite sequences of independent trials of events drawn from the original space. But in that case one has an event whose probability is any desired dyadic rational (for instance "not HHH" has probability $7/8$), so in fact continuity is sufficient. I'm trying to write up all the details in the discrete case; perhaps a similar strategy will work in the continuous case after all. $\endgroup$ Feb 20 '17 at 2:49

Let $X$ be a discrete random variable (r.v.) taking distinct values $x_1,x_2,\dots$, and let $p_i:=P(X = x_i)$. Then the entropy of $X$ is defined by the formula \begin{equation} H(X): = \sum_i p_i \log \frac1{p_i}. \tag{1} \end{equation} Note that it does not matter whatsoever in what set/space the values $x_1,x_2,\dots$ are assumed to be; it does not matter if some of these values are close to, or far away from, one another -- in any sense. What matters is that the $p_i$'s are the probabilities of distinct values of the r.v. $X$. Clearly, if we aggregate some of these values, then the entropy will go down; and if we split some of these values, then the entropy will go up.

Therefore, the formula $H(X) = \int dx\, p(x) \log \frac1{p(x)}$ will hardly make sense if, say, the integral is understood in the Riemann sense, implying the rather arbitrary grouping of the values $x$ according to the standard metric on $\mathbb R$. Moreover and much more importantly, the Riemann sums $\sum_i p(x_i)\Delta x_i \log \frac1{p(x_i)}\,$ for this integral are quite different from the sums $\sum_i p(x_i)\Delta x_i \log\frac1{p(x_i)\Delta x_i}$ that would genuinely correspond to the reality-based definition (1). Also, these latter sums will usually be very large if the $\Delta x_i$'s are very small, and the values of these sums may fluctuate wildly depending on the choice of the $\Delta x_i$'s.

The more general formula $H(X) = \int p(x) \log \frac1{p(x)}\, \mu(dx)$, where $\mu$ is a measure and the grouping of the values $x$ occurs according to the closeness of the corresponding values of $p(x)$ (!!), will hardly make more sense than the Riemann integral.

The only exception here would be when $\mu$ is the counting measure, with which no actual grouping (or splitting) of any values occurs. Then for the density (say $p$) of the distribution of the r.v. $X$ with respect to the counting measure $\mu$, the condition $\int p\,d\mu=1$ can be rewritten as $\sum_x p(x)=1$, which will imply that $p(x)\ne0$ only for (at most) countably many values of $x$, so that the r.v. $X$ is necessarily discrete -- and then we can write \begin{equation*} H(X) = \int p(x) \log \frac1{p(x)}\,\mu(dx) =\sum_x p(x) \log \frac1{p(x)}, \end{equation*} which is the same as (1), up to the change in notation.

So, if the r.v. $X$ is not discrete, then the only reasonable value to assign to the entropy of $X$ appears to be $\infty$, at least from the viewpoint of information theory.

As for the integral $\tilde H(X):=\int p(x) \log \frac1{p(x)}\, \mu(dx)$ in the case when $\mu$ is the Lebesgue measure, the main interest to it seems to be the easily seen fact (see e.g. Barron) that the maximum of $\tilde H(X)$ over all absolutely continuous r.v.'s $X$ with a fixed variance is attained when the distribution of $X$ is normal; moreover, the absolute value of difference of $\tilde H(X)$ from its maximum equals the relative entropy $\int p(x)\log\frac{p(x)}{\varphi(x)}\, dx$, where $\varphi$ is the normal density with the same mean and variance as $p$. However, what is actually used in the proofs is the relative entropy $\int \log\frac{dP}{dQ}\,dP$ (also known as the Kullback--Leibler divergence), which is well defined for any probability measures $P$ and $Q$ such that $P$ is absolutely continuous with respect to $Q$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.