9
$\begingroup$

An ensemble corresponding to a probability distribution usually has finite free energy so it is not a great loss of generality to assume that the ensemble also has finite energy in following discussions.

It is known that when the probability distribution $\mu$(assuming it is also dominated by Lebesgue measure and its density $f_{\mu}(x)$ for simplicity.) has a finite (compact) support then the classical differential entropy $$En(\mu) = \int_{\mathbb{R}} -\log[f_{\mu}(x)]f_{\mu}dx$$ can be bounded by Bekenstein bound (Wiki) and hence finite.

However, when the support of $\mu$ is not finite (compact), there exists counter example that the entropy could be infinite (Math.SE).

Question:

(1)Is finite support of $\mu$ also a necessary condition to make sure the $\mu$ has a finite entropy?

Update: It is clear that a $\mu$ with infinite support can have finite entropy. Thanks Anthony Quas for pointing out.

(2)Is there a characterization(sufficient and necessary condition) of probability distributions/statistical ensembles with finite entropy?

(3)Also, it is known that Bekenstein bound may also be applied(with volumes defined only for atoms) to entropy defined for $\sigma$-algebras(MO.post), so can we translate the characterization in (2) onto $\sigma$-algebras?

(4)From physicists' viewpoint, what will a finite entropy system looks like?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ (1) clearly you could have finite energy but unbounded support by putting a little bit of mass in a small number of places (e.g. if $f$ takes only values 0 and 1, but is not supported on a bounded interval). For (2), the characterization is likely to be that it has finite entropy if and only if it has finite entropy. $\endgroup$
    – Anthony Quas
    Commented Mar 11, 2017 at 23:57
  • $\begingroup$ @AnthonyQuas (1)Oh..yes I updated it.(2) Why is that natural to you? $\endgroup$
    – Henry.L
    Commented Mar 12, 2017 at 0:01
  • 2
    $\begingroup$ My point is that it's unlikely (in my opinion) that there will be a useful characterization of finite entropy that isn't a trivial reformulation of the original criterion. $\endgroup$
    – Anthony Quas
    Commented Mar 12, 2017 at 0:23

1 Answer 1

Reset to default
6
$\begingroup$

A sufficient condition would be that the tails of $f$ decay as $O(x^{-1-\epsilon})$, which is most distributions you might encounter over $\mathbb{R}$.

That being said, the differential entropy of a continuous pdf isn't really a meaningful physical, or information theoretical, quantity. The equation isn't homogeneous: changing the units in which you measure $x$ changes the differential entropy (dimensional analysis can lead to a surprising amount of mathematical insights, even outside of physics).

What does make sense is the KL-divergence of one distribution with respect to another. For a pdf with a compact support, you're implicitly looking at the divergence with respect to the uniform distribution. However, there is no "uniform" distribution over the real line and thus the concept isn't meaningful.

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ The concept is perfectly meaningful. You can take the KL divergence of a measure, not necessarily a probability measure, with respect to another measure as long as the relevant Radon-Nikodym derivative is defined. The uniform distribution over the real line corresponds to Lebesgue measure, and can be used e.g. as an "improper prior" in Bayesian inference. $\endgroup$
    – Qiaochu Yuan
    Commented Mar 21, 2017 at 17:49
  • 3
    $\begingroup$ It's a matter of philosophy. Accepting these priors give you probability paradoxes. $\endgroup$
    – Arthur B
    Commented Mar 21, 2017 at 18:12
  • 3
    $\begingroup$ Also, by that definition, you can get negative entropy, even though the entropy is supposed to be the log of a number of configurations. Yes you can define it mathematically, but how meaningful is it? $\endgroup$
    – Arthur B
    Commented Mar 21, 2017 at 18:21
  • 1
    $\begingroup$ If $f$ is the density of the measure w.r.t. the Lebesgue measure, then this seems pretty restrictive. Is there a necessary condition? $\endgroup$
    – Henry.L
    Commented Mar 21, 2017 at 19:06
  • 1
    $\begingroup$ Entropy is, differential entropy isn't. $\endgroup$
    – Arthur B
    Commented Mar 21, 2017 at 23:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .