# When is the entropy of a $\sigma$-algebra finite?

Let two (countably-generated) $\sigma-$algebras $\mathscr{F,G}$ on the event space $\mathbb{R}$ be given. I believe we also need the atoms of $\mathscr{F,G}$ to be the points of $\mathbb{R}$.

Let $\mu$ be a probability measure on the measurable space $(\mathbb{R},\mathscr{F})$ which is absolutely continuous with respect to Lebesgue measure on $\mathbb{R}$- its density is denoted $f(x)$; define $\mu^{\mathscr{G}}$ to be the restriction of $\mu$ to the sub-measure space $(\mathbb{R}, \mathscr{G})$, and define $\mu^{\mathscr{F}}$ to be the original $\mu$ defined on $(\mathbb{R},\mathscr{F})$.

The (differential) entropy of the countably-generated $\sigma-$algebra $\mathscr{F}$ is equal to or defined to be: $$H_{\mu}(\mathscr{F}) = \int_{\mathbb{R}} -\log[f(x)]\mathrm{d}\mu^{\mathscr{F}}x$$ and that of $\mathscr{G}$ can be defined to be: $$H_{\mu}(\mathscr{G})=\int_{\mathbb{R}}-\log[f(x)]\mathrm{d}\mu^{\mathscr{G}}x$$ To the best of my knowledge, one has that $$\mathscr{G} \subset \mathscr{F} \implies H_{\mu}(\mathscr{G}) \le H_{\mu}(\mathscr{F}) \tag{1}$$ However, this inequality is really only meaningful when $H_{\mu}(\mathscr{G})$ and $H_{\mu}(\mathscr{F})$ are finite.

Question: When are $H_{\mu}(\mathscr{G})$ and $H_{\mu}(\mathscr{F})$ finite?

I have not been able to find much information about the entropy of a sigma-algebra (w.r.t. a given probability measure of course) on the internet, and the four sources [1][2][3][4] which I have been able to find all seem to lack references which discuss the concept clearly.

Note: I am interested in infinite $\sigma$-algebras because of their use in stochastic processes. In particular, I want to be able to say that "larger $\sigma-$algebras contain more information" and to have something like $(1)$ make this statement be true in a rigorous sense. In other words I would like to be able to show that the entropy of a filtration increases with time.

• What do you mean by $\intop (- \log f) d \mu^{\mathscr{G}}$ when $f$ is not $\mathscr{G}$-measurable? – Alexander Shamov Aug 27 '16 at 2:15
• Another issue with your post seems to be that since the reference measure is infinite, some of the conditional measures on the fibers of the partitions $\mathscr{F}$ and $\mathscr{G}$ (are still well-defined, but) may be infinite as well. Thus even if an absolutely continuous measure $\mu$ has an $\mathscr{F}$-measurable density when restricted to $\mathscr{F}$, it doesn't necessarily have a $\mathscr{G}$-measurable density when restricted to $\mathscr{G}$ (think about the trivial $\sigma$-algebra)... – Alexander Shamov Aug 27 '16 at 2:54
• ... The expression $\mathbb{E}[f \mid \mathscr{G}]$ that you suggested would give exactly this density, if you were taking the conditional expectation w.r.t. the normalized conditional Lebesgue measure, which would only make sense if it was "normalizable", i.e. finite. – Alexander Shamov Aug 27 '16 at 2:55
• I do not understand the foundations of your post : "on the event space $\mathbb{R}$ be given. I believe we also need the atoms of $\mathcal F,G$ to be the points of $\mathbb{R}$." It seems that it is ill-typed. Am I right ? could you elaborate a bit ? – Duchamp Gérard H. E. Aug 27 '16 at 6:13
• Disintegration (= conditioning) of probability measures is explained in any decent textbook on probability, see e.g. Kallenberg's "Foundations of modern probability". For the case of $\sigma$-finite measures I don't know of a good reference, but the basic idea is to reduce to the finite measure case: just multiply by some positive $L^1$ density $\phi$ to get an equivalent finite measure, then do the disintegration, then multiply the conditional measures by $\phi^{-1}$. – Alexander Shamov Aug 27 '16 at 15:08