Information theory for uncountably infinite-dimensional continuous random variable

Question

I'm exploring the possibility to apply information theory on an uncountably infinite-dimensional scenario. I found the concept of generalized entropy for continuous random variables defined on finite-dimensional Euclidean space. I wonder whether there are similar concepts for uncountably infinite-dimensional cases. For instance, if I have a distribution on $C([0, 1])$, which is the space of continuous function on $[0, 1]$, then how can I quantify the entropy of this distribution?

Answer 1

You can do this exactly in the same way, except that the right notion is that of relative entropy and that you need a reference measure. Let me explain: on an abstract measurable space $(\Omega,\Sigma)$ choose any reference probability measure $R$. The relative entropy of an arbitrary probability measure $P\in\mathcal P(\Omega)$ with respect to $R$ is then simply $$ H(P|R):=\int_{\Omega}\frac{dP}{dR}(\omega)\log\left(\frac{dP}{dR}(\omega)\right) \,d R(\omega), $$ with the convention is that $0\log 0=0$, that $\frac{dP}{dR}$ denotes the Radon-Nykodim density of $P$ with respect to $R$ if the absolute continuity $P\ll R$ holds, and that $H(P|R):=+\infty$ whenever $P$ is not absolutely continuous w.r.t. $R$. The fact that $R$ is a probability can obviously be relaxed, it can actually be unbounded (but is must still be nonnegative, of course).

Clearly, that $\Omega$ be finite- or infinite-dimensional plays no distinguised role whatsoever in this abstract definition. In my humble opinion people are often misled because in finite dimensions there is a "canonical" reference measure, which is the Lebesgue one $R=dx$. So people often don't realize that $H(\rho)=\int_\Omega \rho(x)\log\rho(x)\,dx$ is actually a relative entropy $H(\rho|dx)$, with a slight abuse of notations that the probability measure $\rho$ and its density $\rho(x)$ w.r.t the Lebesgue measure are identified.

In your particular example $\Omega=C([0,1])$ one possible and usual reference measure is the law of a Brownian motion. The resulting entropy then plays a role sometimes in Girsanov theory and optimal transport, see e.g. this paper or that paper.