An ensemble corresponding to a probability distribution usually has finite free energy so it is not a great loss of generality to assume that the ensemble also has finite energy in following discussions.

It is known that when the probability distribution $\mu$(assuming it is also dominated by Lebesgue measure and its density $f_{\mu}(x)$ for simplicity.) has a finite (compact) support then the classical differential entropy $$En(\mu) = \int_{\mathbb{R}} -\log[f_{\mu}(x)]f_{\mu}dx$$ can be bounded by Bekenstein bound (Wiki) and hence finite.

However, when the support of $\mu$ is not finite (compact), there exists counter example that the entropy could be infinite (Math.SE).


(1)Is finite support of $\mu$ also a necessary condition to make sure the $\mu$ has a finite entropy?

Update: It is clear that a $\mu$ with infinite support can have finite entropy. Thanks Anthony Quas for pointing out.

(2)Is there a characterization(sufficient and necessary condition) of probability distributions/statistical ensembles with finite entropy?

(3)Also, it is known that Bekenstein bound may also be applied(with volumes defined only for atoms) to entropy defined for $\sigma$-algebras(MO.post), so can we translate the characterization in (2) onto $\sigma$-algebras?

(4)From physicists' viewpoint, what will a finite entropy system looks like?

  • 1
    $\begingroup$ (1) clearly you could have finite energy but unbounded support by putting a little bit of mass in a small number of places (e.g. if $f$ takes only values 0 and 1, but is not supported on a bounded interval). For (2), the characterization is likely to be that it has finite entropy if and only if it has finite entropy. $\endgroup$ – Anthony Quas Mar 11 '17 at 23:57
  • $\begingroup$ @AnthonyQuas (1)Oh..yes I updated it.(2) Why is that natural to you? $\endgroup$ – Henry.L Mar 12 '17 at 0:01
  • 2
    $\begingroup$ My point is that it's unlikely (in my opinion) that there will be a useful characterization of finite entropy that isn't a trivial reformulation of the original criterion. $\endgroup$ – Anthony Quas Mar 12 '17 at 0:23

A sufficient condition would be that the tails of $f$ decay as $O(x^{-1-\epsilon})$, which is most distributions you might encounter over $\mathbb{R}$.

That being said, the differential entropy of a continuous pdf isn't really a meaningful physical, or information theoretical, quantity. The equation isn't homogeneous: changing the units in which you measure $x$ changes the differential entropy (dimensional analysis can lead to a surprising amount of mathematical insights, even outside of physics).

What does make sense is the KL-divergence of one distribution with respect to another. For a pdf with a compact support, you're implicitly looking at the divergence with respect to the uniform distribution. However, there is no "uniform" distribution over the real line and thus the concept isn't meaningful.

  • 1
    $\begingroup$ The concept is perfectly meaningful. You can take the KL divergence of a measure, not necessarily a probability measure, with respect to another measure as long as the relevant Radon-Nikodym derivative is defined. The uniform distribution over the real line corresponds to Lebesgue measure, and can be used e.g. as an "improper prior" in Bayesian inference. $\endgroup$ – Qiaochu Yuan Mar 21 '17 at 17:49
  • 3
    $\begingroup$ It's a matter of philosophy. Accepting these priors give you probability paradoxes. $\endgroup$ – Arthur B Mar 21 '17 at 18:12
  • 3
    $\begingroup$ Also, by that definition, you can get negative entropy, even though the entropy is supposed to be the log of a number of configurations. Yes you can define it mathematically, but how meaningful is it? $\endgroup$ – Arthur B Mar 21 '17 at 18:21
  • 1
    $\begingroup$ If $f$ is the density of the measure w.r.t. the Lebesgue measure, then this seems pretty restrictive. Is there a necessary condition? $\endgroup$ – Henry.L Mar 21 '17 at 19:06
  • 1
    $\begingroup$ Entropy is, differential entropy isn't. $\endgroup$ – Arthur B Mar 21 '17 at 23:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.