entropy and flatness of densities

Question

I was reading C.R Rao's Linear Statistical inference. Rao presents the entropy of a continuous distribution (expectation of -log density) as a measure of closeness to the uniform distribution, and shows that of all distributions whose densities which have a bounded interval [a,b] as its support the uniform distribution on [a,b] has the highest entropy. Can this connection be formalized? For example if I restrict myself to probability distributions which put all mass on a bounded interval and have sufficiently smooth densities, is there a measure of the flatness of the density based on geometry which is connected to the entropy of the distribution.

Answer 1

Since entropy distance between two probability measures bounds total variation distance, and since total variation distance is basically the $L^1$ distance of density functions, my guess is that entropy difference does not detect small but sharp fluctuations in the density function. So for instance one can cook up a family of density functions developing a singularity at one point, but whose entropy converges to that of the uniform (simply construct a rectangular peak with increasing height and decreasing width, with area held fixed). However one can probably think of entropy as the "$L^{\log}$"-norm, so in that sense it does measure the flatness of the density. But maybe that's too obvious.

Answer 2

I really like the answer of John Jiang, I just want to give an answer by solving the optimization problem and discussing when the solution exists.

In the discrete case it is easy to notice that

$$\max_{\sum_{i=1}^n p_i=1, \forall p_i\in[0,1]}-\sum_i p_i\log (p_i)$$

is obtained with for all $i=1,\dots,n$, $p_i=1/n$. To show this there are two alternatives:

1. Use the KKT condition (brutal force)
2. Use symmetry (elegant ... )

I prefer the second alternative: First, notice that the function to maximize is continuous and strictly concave and that the optimization domain is compact convex, hence a unique solution, say $p^*=(p_1^*,\dots,p_n^*)$. For $i,j=1,\dots,n$, let $T_{ij}$ be the application that swaps $i$ and $j$ coordinates of a vector of $\mathbb{R}^n$. Obviously our optimization problem is invariant by the use of all these applications (group of permutations…) and $T_{ij} (p^* )=p^* $ for all $i,j$ (because $p^*$ is the unique solution and $T_{ij}(p^*)$ is also a solution. This gives you the solution since $p_1=\dots=p_n$. An interpretation of this result is that $p^*=(1/n,\dots,1/n)$ (the uniform measure) is the haar measure of the group of translations on the torus $\{1,\dots,n\}$.

The question you ask is whether this result can be extended in the continuous case. As mentioned by John Jiang, the answer is no. There is something that is still true: If you consider the optimization problem $$ \sup_{f\geq 0 \int_0^1 f=1} -\int_0^1 f\log(f) dx $$ and if the solution exists and is unique, then it is the uniform distribution that is the solution. To see this you need to transform a little bit the problem by concidering $[0,1]$ as a torus and apply translations… the haar measure of the group of translations on $[0,1]$ (see http://en.wikipedia.org/wiki/Haar_measure#Examples) is the lebesgue… Hence, if you can keep the compact convex assumption (for a given topology) and the continuity assumption (for the function you minimize) … you’ll get a positive answer. You need to introduce topology (compacity is lost with the example of John Jiang (at least for a topology that keeps the function to optimize continuous (or lower semi-continuous)) !

Answer 3

Two great answers above - I just wanted to comment on the "measure of closeness to the uniform distribution"-part. Now I do not have Rao's book in my office so I can't check if this is already in it, my apologies if that's the case.

There is a concept in information theory called the relative entropy, or Kullback-Leibler divergence, which measures the distance between two probability distributions Wiki: Kullback-Leibler.

Since we are now only interested in continuous distributions, let $P$ and $Q$ be two such distributions with densities $p$ and $q$ respectively. The Kullback-Leibler divergence from $P$ to $Q$ is $\mathcal{H}(P|Q) = \int \log \frac{dP}{dQ}dP = \int \log \frac{p(x)}{q(x)}p(x)dx $

Now take p to have support $[a,b]$ and let $Q$ be the uniform distribution on this interval (i.e., $q = \frac{1}{b-a}$ on $[a,b]$). Inserting this into the expression for $\mathcal{H}(P|Q)$ yields

$\mathcal{H}(P|Q) = \int \log p(x) dP(x) + \log(b-a), $

where the first term is the entropy of $P$ sans the minus sign. Thus, since the Kullback-Leibler divergence in this case is precisely a measure of the closeness of $P$ to the uniform distribution $Q$, the entropy of $P$ will have this interpretation - the term $\log (b-a)$ is only a constant and will be present for every distribution $P$. It holds that $\mathcal{H}(P|Q) = 0$ iff $P=Q$ and this explains why the uniform distribution will also be the one with the highest entropy, namely $\log (b-a)$