The entropy of a partition of unity

Question

A partition of unity can be thought of as a blurred out set partition: identify the fibers of the set partition with their indicator functions; the sum of these indicators is $1$. I would like to extend the definition of entropy from set partitions to partitions of unity. My purpose is to capture the notion of information obtained by making noisy or imprecise measurements (I view ordinary set partitions as measurements that are quantized but noiseless). Also, this setup is a satisfying formulation of the idea of a coarse-graining in statistical physics.

Before giving the definitions, here is my question: Do these generalized partitions give rise to equivalent results and definitions in the theory of dynamical systems? For example, is it possible to modify the definition of the dynamical (Kolmogorov–Sinai) entropy to allow partitions of unity, and is the resulting definition equivalent? What about the Shannon–McMillan–Breiman Theorem? More broadly, do they have any interesting new properties at all? A negative answer would be interesting too, because it would mean that we aren't losing anything by working only with deterministic measurements. In addition, I'd be very interested to hear of any existing work or connections to other areas.

Definitions:

Given a probability space $(X, \mu)$, a (finite, labeled) set partition of $X$ is a finite indexed family of nonempty pairwise disjoint measurable sets whose union is $X$. Given a partition $\Pi$, the information function is the random variable $I_\Pi : X \to \mathbb R, \;\; x \mapsto -\log \mu \,E_x$ where $E_x$ is the unique element of $\Pi$ such that $x \in E_x$. The static (or Shannon) entropy of a set partition $\Pi$ is the expectation of the information function, $H(\Pi) := \mathbb E(I_\Pi) = -\sum_{E \in \Pi} \mu E \; \log \mu E$.

More generally, we can define a (measure-theoretic, finite) partition of unity on $X$ to be an indexed family of measurable functions $f_1, \ldots, f_n : X \to [0, 1]$, with $\sum_i f_i(x) = 1$ for every $x \in X$. Given a function $f : X \to \mathbb R_{\ge 0}$ whose integral is strictly positive, let $$I(f) := \int \hat f(x) \, \log \hat f(x) \, \mathrm{d}\mu(x) \quad \text{where}\; \hat f = \frac{f} {\int f\,\mathrm{d}\mu}.$$ (There's no minus sign: not a typo. Heuristically speaking, $I(f)$ is the information obtained about $x$ by observing the event "$f$ occured"; cf. the definition of differential entropy.)

A partition of unity is essentially the same thing as a probability transition kernel $K$ from $X$ into $\{1, \ldots, n\}$: The measure $K(x, \cdot)$ on $\{1, \ldots n\}$ is the aforementioned noisy measurement of $x \in X$. Let $\Pi$ be a partition of unity and $K$ the associated kernel. Let the static entropy of $\Pi$ be the expectation of the pullback $K I$. Equivalently, $$H(\Pi) \;:=\; \sum_{f \in \Pi} \left( \int f \mathrm{d}\mu \right) I(f).$$

Here are a few trivial properties, which the definitions above were meant to capture:

1. The two definitions of $H$ are compatible: If $\Pi$ is the family of indicator functions of the fibers of a set partition, we recover the static entropy of the partition. This is because $I(\mathbf 1_E) = -\log \mu E$ for every measurable set $E \subseteq X$.
2. $I(kf) = I(f)$ for every $k > 0$ and $f : X \to \mathbb R_{\ge 0}$ with $\int f \mathrm{d}\mu > 0$.
3. $I(f) = 0$ whenever $f \equiv c > 0$. (A flat posterior density contains no information.)
4. If $X = \mathbb R^d$ and $A$ is an invertible linear operator, then $I(f \circ A) = I(f) + \log |\det A|$ for every $f : X \to \mathbb R_{\ge 0}$ with $\int f \mathrm{d}\mu > 0$.

A few more definitions will need to be generalized, though I haven't worked out the rest yet. For example, one candidate for the join of two partitions of unity $\Pi_1, \Pi_2$ is the family of pointwise products, $(f_1 f_2)_{f_1 \in \Pi_1, f_2 \in \Pi_2}$ (this is a partition of unity because $\sum_{i,j} (f_{1i} f_{2j})(x)$ $=$ $\sum_i f_{1i}(x) \sum_j f_{2j}(x)$ $= 1$).

Answer 1

In the parlance of information theory, your functional $I$ is a conditional relative entropy, and $H$ is a mutual information. Indeed, $\mu$ and $K$ together define a joint probability measure on $X \times \{1,\dots, n\}$. Let $(W,N)$ be a random vector on $X \times \{1,\dots, n\}$ with that law (i.e. $W\sim\mu$ and $N|\{W=x\}\sim K(x,\cdot)$). Then $H(\Pi)$ in your notation is the mutual information $I(W;N)$ (in traditional notation of Shannon theory). Also, $I(f_n) = D(P_{W|N=n}||P_W)$, where $P_{W}$ and $P_{W|N=n}$ denote the laws of $W$ and $W$ conditioned on $\{N=n\}$, respectively, and $D(\cdot||\cdot)$ is the relative entropy.

Edited to add: Recognizing the quantity $H$ as a mutual information highlights your difficulty in extending your definition to the join of two partitions. If you've got two partitions $\Pi_1$ and $\Pi_2$, then in analogy to my response above, these together with $\mu$ define random variables $W,N_1,N_2$, but only specify joint laws of the couples $(W,N_1)$ and $(W,N_2)$. The thing you'd likely be after is something like the mutual information $I(W; N_1,N_2)$, but this requires you to provide additional information to specify the joint law of the triple $(W,N_1,N_2)$. In any case, you can search on the term 'mutual information' to see that it agrees with your $H$ and has many known and useful properties (including those you enumerate). Further, it indeed captures the notion of information obtained by making noisy or imprecise measurements, consistent with your motivation.