# Is there an axiomatic characterization of the entropy of a continuous random variable?

Let $X$ be a random variable taking values in $\{1,\ldots,n\}$, and let $p_i$ denote the probability of the event $\{X = i\}$. Shannon defined the entropy of $X$ to be the quantity

$$H(X) = -\sum_i p_i \log p_i$$

with the convention $x \log x = 0$. This definition is the starting point for all of information theory, and consequently it has been provided with numerous axiomatic characterizations. Shannon gave a characterization in his original paper; Myron Tribus gave another involving how "surprising" a random variable is on average; yet another approach constructs entropy as the objective function for certain optimization problems (turning the principle of maximum entropy into a definition); and Baez, Fritz, and Leinster gave still another characterization involving convexity and functorial properties. I'm sure this list is not exhaustive.

But I have only ever seen axiomatic characterizations for discrete random variables. Shannon himself defined the entropy of a continuous random variable by:

$$H(X) = -\int p(x) \log p(x)\, dx$$

where $p(x)$ is the density function of $X$. Jaynes argued here that this is the wrong definition because it has the wrong units and it transforms incorrectly under a change of coordinates; he was able to modify the definition accordingly by taking a limit of discrete entropies. Regardless of which definition is correct my question is this:

Is there an axiomatic characterization of the entropy of a continuous random variable which generalizes a corresponding characterization in the discrete case? If so, what is the "right" class of measure spaces to which this characterization applies?

• Not knowing enough to make this an answer, - there is fundamental work by Coleman and Owen "A mathematical foundation for thermodynamics" where they do, I believe, what you are looking for. I could be able to make an answer about a kind of categorification of their work that Lawvere made in his work on state categories and semicontinuous entropy functions Commented Feb 19, 2017 at 11:15
• I'll chase down these references - thanks! Commented Feb 19, 2017 at 15:01
• Do you want to wrap up this matter (over 6 years after your post)? Commented Jun 21, 2023 at 14:26
• @IosifPinelis I'm not quite satisfied that we have an answer just yet - I'm looking for something of the form "Differential entropy is the unique function on the space X with properties A, B, and C". I would even be satisfied with a characterization of that form for relative entropy, so long as it applies to continuous random variables. Commented Jul 15, 2023 at 15:43
• @PaulSiegel : I have added the details. Commented Jul 16, 2023 at 10:57

$$\newcommand{\Si}{\Sigma}\newcommand{\PP}{\mathscr P}\newcommand{\PPP}{\mathfrak P}$$Let $$X$$ be a discrete random variable (r.v.) taking distinct values $$x_1,x_2,\dots$$, and let $$p_i:=P(X = x_i)$$. Then the entropy of $$X$$ is defined by the formula $$\begin{equation*} H(X): = \sum_i p_i \log \frac1{p_i}. \tag{1} \end{equation*}$$ Note that it does not matter whatsoever in what set/space the values $$x_1,x_2,\dots$$ are assumed to be; it does not matter if some of these values are close to, or far away from, one another -- in any sense. What matters is that the $$p_i$$'s are the probabilities of distinct values of the r.v. $$X$$. Clearly, if we aggregate some of these values, then the entropy will go down; and if we split some of these values, then the entropy will go up.

Therefore, the formula $$H(X) = \int dx\, p(x) \log \frac1{p(x)}$$ will hardly make sense if, say, the integral is understood in the Riemann sense, implying the rather arbitrary grouping of the values $$x$$ according to the standard metric on $$\mathbb R$$. Moreover and much more importantly, the Riemann sums $$\sum_i p(x_i)\Delta x_i \log \frac1{p(x_i)}\,$$ for this integral are quite different from the sums $$\sum_i p(x_i)\Delta x_i \log\frac1{p(x_i)\Delta x_i}$$ that would genuinely correspond to the reality-based definition (1). Also, these latter sums will usually be very large if the $$\Delta x_i$$'s are very small, and the values of these sums may fluctuate wildly depending on the choice of the $$\Delta x_i$$'s.

The more general formula $$H(X) = \int p(x) \log \frac1{p(x)}\, \mu(dx)$$, where $$\mu$$ is a measure and the grouping of the values $$x$$ occurs according to the closeness of the corresponding values of $$p(x)$$ (!!), will hardly make more sense than the Riemann integral.

The only exception here would be when $$\mu$$ is the counting measure, with which no actual grouping (or splitting) of any values occurs. Then for the density (say $$p$$) of the distribution of the r.v. $$X$$ with respect to the counting measure $$\mu$$, the condition $$\int p\,d\mu=1$$ can be rewritten as $$\sum_x p(x)=1$$, which will imply that $$p(x)\ne0$$ only for (at most) countably many values of $$x$$, so that the r.v. $$X$$ is necessarily discrete -- and then we can write $$\begin{equation*} H(X) = \int p(x) \log \frac1{p(x)}\,\mu(dx) =\sum_x p(x) \log \frac1{p(x)}, \end{equation*}$$ which is the same as (1), up to the change in notation.

So, if the r.v. $$X$$ is not discrete, then the only reasonable value to assign to the entropy of $$X$$ appears to be $$\infty$$, at least from the viewpoint of information theory.

As for the integral $$\tilde H(X):=\int p(x) \log \frac1{p(x)}\, \mu(dx)$$ in the case when $$\mu$$ is the Lebesgue measure, the main interest to it seems to be the easily seen fact (see e.g. Barron) that the maximum of $$\tilde H(X)$$ over all absolutely continuous r.v.'s $$X$$ with a fixed variance is attained when the distribution of $$X$$ is normal; moreover, the absolute value of difference of $$\tilde H(X)$$ from its maximum equals the relative entropy $$\int p(x)\log\frac{p(x)}{\varphi(x)}\, dx$$, where $$\varphi$$ is the normal density with the same mean and variance as $$p$$.

However, what is actually used in the proofs is the relative entropy $$D(P\parallel Q):=\int_\Omega dP\,\log\frac{dP}{dQ}$$ (also known as the Kullback--Leibler divergence), which is well defined for any probability measures $$P$$ and $$Q$$ such that $$P$$ is absolutely continuous with respect to $$Q$$. One can show that (with appropriate conventions about the division and multiplication by $$0$$) $$\begin{equation*} D(P\parallel Q)=\sup\sum_j P(A_j)\log\frac{P(A_j)}{Q(A_j)} =\lim\sum_j P(A_j)\log\frac{P(A_j)}{Q(A_j)},\tag{2}\label{2} \end{equation*}$$ where the $$\sup$$ is taken over the set (say $$\mathcal P$$) of all finite (or, alternatively, at most countable) measurable partitions $$(A_j)$$ of $$\Omega$$ and the $$\lim$$ is taken over the set $$\mathcal P$$ of partitions directed by the refinement relation. (Please let me know if details on \eqref{2} are needed). So, the notion of the relative entropy in general reduces to this notion for discrete distributions, for which axiomatic approaches are well known.

Added on 7/16/23: This can be done as follows. Assume the following axioms:

s1: The (measure of) surprise contained in an event is nonnegative and depends only of the probability of the event. That is, there is a function $$f\colon[0,1]\to[0,\infty]$$ such that for any probability space $$(S,\Si)$$, any probability measure $$P$$ on $$\Si$$, and any $$A\in\Si$$ the surprise $$s_P(A)$$ contained in the event $$A$$ with respect to (w.r.t.) the probability measure $$P$$ is $$f(P(A))$$. Let us refer to $$s_P(A)$$ as the $$P$$-surprise of $$A$$.

s2: The function $$f$$ is measurable and nontrivial (that is, not everywhere zero and not everywhere $$\infty$$).

s3: If two independent events occur, their $$P$$-surprises get added: If events $$A$$ and $$B$$ are independent w.r.t. a probability measure $$P$$, then $$s_P(A\cap B)=s_P(A)+s_P(B)$$.

rs1: For any probability space $$(S,\Si)$$, any probability measures $$P$$ and $$Q$$ on $$\Si$$, and any $$A\in\Si$$, the relative $$(P\parallel Q)$$-surprise $$s_{P\parallel Q}(A)$$ of $$A$$ is the difference $$s_Q(A)-s_P(A)$$.

rs2: For any probability space $$(S,\Si)$$, any probability measures $$P$$ and $$Q$$ on $$\Si$$, and any $$\Si$$-measurable partition $$\PP=(A_1,\dots,A_n)$$ of $$S$$, the $$(P\parallel Q)$$-surprise of $$\PP$$, denoted by $$s_{P\parallel Q}(\PP)$$, is the (discrete) random variable (r.v.) that is the simple function $$\sum_{j=1}^n s_{P\parallel Q}(A_j)\,1_{A_j}$$.

rs3: For any probability space $$(S,\Si)$$, any probability measures $$P$$ and $$Q$$ on $$\Si$$, and any $$\Si$$-measurable partition $$\PP=(A_1,\dots,A_n)$$ of $$S$$, the entropy $$D_\PP(P\parallel Q)$$ of $$P$$ relative to $$Q$$ w.r.t. $$\PP$$ is the expectation w.r.t. to $$P$$ of the r.v. $$s_{P\parallel Q}(\PP)$$ (that is the $$(P\parallel Q)$$-surprise of the partition $$\PP$$): $$\begin{equation*} D_\PP(P\parallel Q):=Es_{P\parallel Q}(\PP). \end{equation*}$$ (So, $$D_\PP(P\parallel Q)$$ is the average additional surprise w.r.t. the partition $$\PP$$ that one will get assuming that the probability distribution is $$Q$$ whereas the actual probability distribution is $$P$$.)

rs4: For any probability space $$(S,\Si)$$ and any probability measures $$P$$ and $$Q$$ on $$\Si$$, the entropy $$D_\PP(P\parallel Q)$$ of $$P$$ relative to $$Q$$ is the supremum of the entropy $$D_\PP(P\parallel Q)$$ of $$P$$ relative to $$Q$$ w.r.t. $$\PP$$ over the set (say $$\PPP$$) of all finite $$\Si$$-measurable partitions $$\PP$$ of $$S$$: $$\begin{equation*} D(P\parallel Q):=\sup_{\PP\in\PPP}D_\PP(P\parallel Q). \end{equation*}$$

It follows easily from Axioms s1, s2, s3 that $$f(p)=\log_b\frac1p$$ for some real $$b>1$$ and all $$p\in(0,1]$$, with $$f(0)=\infty$$. So, by Axioms rs1, rs2, rs3, for any probability space $$(S,\Si)$$, any probability measures $$P$$ and $$Q$$ on $$\Si$$, and any $$\Si$$-measurable partition $$\PP=(A_1,\dots,A_n)$$ of $$S$$, $$\begin{equation*} D_\PP(P\parallel Q)=\sum_{j=1}^n P(A_j)\log_b\frac{P(A_j)}{Q(A_j)}, \tag{3}\label{3} \end{equation*}$$ assuming the conventions $$p\log_b\frac pq:=0$$ whenever $$p=0$$ and $$p\log_b\frac pq:=\infty$$ whenever $$p>0=q$$.

Then it is easy to see that $$D_{\PP_1}(P\parallel Q)\le D_{\PP_2}(P\parallel Q)$$ if a partition $$\PP_2$$ is a refinement of a partition $$\PP_1$$. Now, with the set $$\PPP$$ directed by refinement, one can show that $$\begin{equation*} D(P\parallel Q) =\lim_{\PP\in\PPP}D_\PP(P\parallel Q)=\int_S dP\,\log_b\frac{dP}{dQ}; \end{equation*}$$ the latter integral is defined as $$\infty$$ if $$P$$ is not absolutely continuous w.r.t. $$Q$$.

That is, if $$\mu$$ is any measure w.r.t. which $$P$$ and $$Q$$ are absolutely continuous (for instance, one may take $$\mu=P+Q$$) with respective densities $$p$$ and $$q$$, then $$\begin{equation*} D(P\parallel Q)=\int_S d\mu\,p\log_b\frac pq. \quad\Box \end{equation*}$$

How about using Tribus' surprise-based characterization again? After all if $X$ and $Y$ are independent random variables with pdfs $f_X$, $f_Y$, then the joint density $f_{X,Y}(x,y)$ is $f_X(x)f_Y(y)$ and so taking log converts it into a sum, as in the discrete argument.

• This idea (as expressed, say, here: en.wikipedia.org/wiki/Self-information) is quite compelling, but as I tried to work it out in the continuous case I realized that I had doubts even about the discrete case. Consider the fair coin flip sample space $\{H, T\}$ with P(H) = P(T) = 1/2. The event $\{H, T\}$ is independent of $H$ and $T$, as is the event $\emptyset$, but monotonicity and independence only give $S(\{H, T\}) = 0$ and $S(\emptyset) = \infty$. There are no additional pairs of independent events to fix the values of $S(H)$ and $S(T)$, so it seems any positive value will do. Commented Feb 19, 2017 at 14:55
• Well but $S (H)=S (T)$ since it's a fair coin. And then an arbitrary positive value is of course fixed to equal 1... Commented Feb 19, 2017 at 16:59
• Why $1$? The monotonicity condition forces $0 < S(H) = S(T) < \infty$, but there are no tools with which the additivity condition can narrow it down further. Of course $S$ is only well defined up to a constant, so my example wasn't very good; let's modify it so that $P(H) = 1/4$ and $P(T) = 3/4$. Then we expect $S(H) = \log 4$ and $S(T) = \log(4/3)$, but where do we run afoul of the axioms if we instead set $S(H) = \log 4$ and $S(T) = \log 2$ for instance? Commented Feb 19, 2017 at 17:36
• I suppose once you have the surprise of an event of probability $1/2^n$ for each $n$ then adding the assumption that the surprise function is real analytic you can deduce the value at other points. Commented Feb 19, 2017 at 22:50
• Obtaining events of probability $1/2^n$ already forces one to define $S$ on a new probability space whose points correspond to finite sequences of independent trials of events drawn from the original space. But in that case one has an event whose probability is any desired dyadic rational (for instance "not HHH" has probability $7/8$), so in fact continuity is sufficient. I'm trying to write up all the details in the discrete case; perhaps a similar strategy will work in the continuous case after all. Commented Feb 20, 2017 at 2:49