Many commentators (e.g. Jaynes, Rota) argue that the notion of "differential entropy" is problematic (as commonly defined by $ h(X) = \int ( \log\frac{1}{p(x)} ) p(x) \, dx $, where $X$ is a random variable with a probability density function $p$). Differential entropy has several often-mentioned deficiencies, in contrast to discrete entropy:

- Its values are not always nonnegative.
- It is not invariant with respect to change of variables.
- It assumes additional structure: a particular underlying measure, and the existence of the density $p$.

These problems all essentially arise because:

- It is not derived from information-theoretic first principles; it is merely defined by analogy to $\sum p_i \log\frac{1}{p_i}$.

This evidence suggests that differential entropy does not have much intrinsic significance, if at all. In particular, ideas like the "principle of maximum entropy", whose foundations lie in the information-theoretic role of discrete entropy, should not blindly generalize to differential entropy.

On the other hand, there are major results about differential entropy which seem to say the opposite. For example, it is well-known that, among all distributions with a particular mean and variance, the Gaussian distribution has the greatest differential entropy. Artstein, Ball, Barthe, and Naor have even proven that the differential entropy of a normalized sum of i.i.d. random variables is monotonically increasing, showing that the central limit theorem has behavior similar to the second law of thermodynamics. Maximum entropy ideas seem to carry over from discrete to differential entropy just fine!

Another example is a form of the uncertainty principle (from quantum mechanics and/or Fourier analysis) expressed in terms of differential entropy (see Wikipedia). This form is stronger than the traditional formulation involving standard deviation. Here, differential entropy appears to fill the role of a respectable measure of uncertainty.

Hence, confusion. How can I reconcile these conflicting bodies of evidence? Should the complaints about differential entropy be dismissed? Or are the nontrivial results about differential entropy somehow not meaningful, except perhaps as shadows of something more natural? Is differential entropy really conceptually significant?

Your insight is appreciated.