KL-divergence and sub-$\sigma$-algebras

Question

I am trying to understand if the following claim is true:

Let $P$, $Q$ be probability measures on $\mathcal{X}$. For any $\sigma$-algebra $\mathcal{G}$, with countably many atoms (sets with $\emptyset$ as their only subset in $\mathcal{G}$), let $$D(P||Q|\mathcal{G}) = \sum_{A\in atom(\mathcal{G})} P(A)\log\frac{P(A)}{Q(A)}.$$ Then for any $\sigma$-algebra $\mathcal{H} \subset \mathcal{G}$ we have $D(P||Q|\mathcal{H}) \leq D(P||Q|\mathcal{G})$.

First of all, I think there should be some assumption regarding the measurability of atoms of $\mathcal{G}$ with respect to $P$ and $Q$.

Apart from that, I cannot see why the inequality would necessarily holds. To build a counterexample, let $\lambda$ be the counting measure on $\{0\}$ and let $(\mathcal{Y},\mathcal{F},\mu)$ be a probability space with an atomless $\sigma$-algebra. For $\alpha\in[0,1]$, There is a probability space on $\mathcal{X} = Y\cup \{0\}$ with probability measure $a\mu' + (1-a)\lambda'$, where the prime measures act as $\mu'(A) = \mu(A\cap \mathcal{Y})$ and $\lambda'(A) = \lambda(A\cap \{0\})$. Take $\mathcal{G}$ to be the $\sigma$-algebra generated in $\mathcal{X}$ by $\mathcal{F}$ and let $\mathcal{H} = \{\emptyset,\mathcal{Y},\{0\},\mathcal{X}\}$. Now $\mathcal{H} \subset \mathcal{G}$ and they both contain finitely many atoms, but we have $$D\left(\frac{1}{2}\mu' + \frac{1}{2}\lambda'||a\mu' + (1-a)\lambda'|\mathcal{G}\right) - D\left(\frac{1}{2}\mu' + \frac{1}{2}\lambda'||a\mu' + (1-a)\lambda'|\mathcal{H}\right) = -\frac{1}{2}\log\left(\frac{a}{2}\right)$$ which is $\leq0$ as long as $a\geq$2.

Having said that, if I assume $\mathcal{X} = \cup atom(\mathcal{G})$ in the original statement, this argument would collapse and the inequality could hold.

Is there a flaw in my argument? I have assumed the existence of measures with various properties (most notably a probability measure with an atomless sigma algebra), but if I remember my measure theoretic basics correctly, all of these existences can be proven.

Answer 1

$\newcommand{\G}{\mathcal G}\newcommand{\HH}{\mathcal H}\newcommand{\A}{\mathcal A}\newcommand{\X}{\mathcal X}$First here, a nonempty member $A$ of a sigma-algebra $\G$ is an atom of $\G$ if the only proper subset of $A$ in $\G$ is empty.

Let now $\A(\G)$ denote the set of all atoms of $\G$. Without the condition \begin{equation*} \X=\cup\A(\G),\tag{1} \end{equation*} it is of course easy to construct counterexamples to the inequality \begin{equation*} D(P||Q|\HH)\le D(P||Q|\G),\tag{2} \end{equation*} as was done in your post.

On the other hand, assuming that (1) holds and that $\A(\G)$ is (at most) countable, inequality (2) holds. Indeed, then every atom $B$ of $\HH$ is a disjoint countable union of atoms of $\G$: \begin{equation*} B=\cup\A_B(\G), \end{equation*} where \begin{equation*} \A_B(\G):=\{A\in\A(\G)\colon A\subseteq B\}. \end{equation*} So, \begin{align*} D(P||Q|\G)&=\sum_{A\in\A(\G)}P(A)\log\frac{P(A)}{Q(A)} \\ &=\sum_{B\in\A(\HH)}\sum_{A\in\A_B(\G)}P(A)\log\frac{P(A)}{Q(A)}, \end{align*} \begin{align*} D(P||Q|\HH)&=\sum_{B\in\A(\HH)}P(B)\log\frac{P(B)}{Q(B)} \\ & =\sum_{B\in\A(\HH)}\sum_{A\in\A_B(\G)}P(A)\log\frac{P(B)}{Q(B)}, \end{align*} \begin{align*} D(P||Q|\G)-D(P||Q|\HH) &=\sum_{B\in\A(\HH)}\sum_{A\in\A_B(\G)}P(A)\log\frac{P(A|B)}{Q(A|B)} \\ &=\sum_{B\in\A(\HH)}P(B)\sum_{A\in\A_B(\G)}P(A|B)\log\frac{P(A|B)}{Q(A|B)} \\ &=\sum_{B\in\A(\HH)}P(B)D\big(P(\cdot|B)||Q(\cdot|B)\big)\ge0, \end{align*} since $D\big(P(\cdot|B)||Q(\cdot|B)\big):=\sum_{A\in\A_B(\G)}P(A|B)\log\frac{P(A|B)}{Q(A|B)}\ge0$.

Thus, (2) holds.