# A necessary condition for differential entropy to be finite

An ensemble corresponding to a probability distribution usually has finite free energy so it is not a great loss of generality to assume that the ensemble also has finite energy in following discussions.

It is known that when the probability distribution $\mu$(assuming it is also dominated by Lebesgue measure and its density $f_{\mu}(x)$ for simplicity.) has a finite (compact) support then the classical differential entropy $$En(\mu) = \int_{\mathbb{R}} -\log[f_{\mu}(x)]f_{\mu}dx$$ can be bounded by Bekenstein bound (Wiki) and hence finite.

However, when the support of $\mu$ is not finite (compact), there exists counter example that the entropy could be infinite (Math.SE).

Question:

(1)Is finite support of $\mu$ also a necessary condition to make sure the $\mu$ has a finite entropy?

Update: It is clear that a $\mu$ with infinite support can have finite entropy. Thanks Anthony Quas for pointing out.

(2)Is there a characterization(sufficient and necessary condition) of probability distributions/statistical ensembles with finite entropy?

(3)Also, it is known that Bekenstein bound may also be applied(with volumes defined only for atoms) to entropy defined for $\sigma$-algebras(MO.post), so can we translate the characterization in (2) onto $\sigma$-algebras?

(4)From physicists' viewpoint, what will a finite entropy system looks like?

• (1) clearly you could have finite energy but unbounded support by putting a little bit of mass in a small number of places (e.g. if $f$ takes only values 0 and 1, but is not supported on a bounded interval). For (2), the characterization is likely to be that it has finite entropy if and only if it has finite entropy. – Anthony Quas Mar 11 '17 at 23:57
• @AnthonyQuas (1)Oh..yes I updated it.(2) Why is that natural to you? – Henry.L Mar 12 '17 at 0:01
• My point is that it's unlikely (in my opinion) that there will be a useful characterization of finite entropy that isn't a trivial reformulation of the original criterion. – Anthony Quas Mar 12 '17 at 0:23

A sufficient condition would be that the tails of $f$ decay as $O(x^{-1-\epsilon})$, which is most distributions you might encounter over $\mathbb{R}$.
That being said, the differential entropy of a continuous pdf isn't really a meaningful physical, or information theoretical, quantity. The equation isn't homogeneous: changing the units in which you measure $x$ changes the differential entropy (dimensional analysis can lead to a surprising amount of mathematical insights, even outside of physics).
• If $f$ is the density of the measure w.r.t. the Lebesgue measure, then this seems pretty restrictive. Is there a necessary condition? – Henry.L Mar 21 '17 at 19:06