3
$\begingroup$

The question concerns a very general setting and a very general inequality about KL divergence. I'm writing this thread to verify whether my intuition is correct.

Let $E_1, E_2$ be two measurable spaces, $f, g: E_1 \rightarrow E_2$ two measurable functions, and $A, B$ two random variables taking values in $E_1$. Is the following inequality:

$D_{KL}(f(A) || g(B)) \le D_{KL}(A || B)$

always true?

In case it isn't, I'd like to ask whether the following inequality is always true:

$D_{KL}(f(A) || f(B)) \le D_{KL}(A || B)$

$\endgroup$

2 Answers 2

3
$\begingroup$

$\newcommand{\si}{\sigma}\newcommand{\F}{\mathcal F}\newcommand{\G}{\mathcal G}\newcommand{\pa}{\parallel}$The first inequality is obviously false in general, e.g. when $A=B$ but $f(A)$ differs from $g(B)=g(A)$ in distribution.

The second inequality is true. A bit more generally, let $\mu$ and $\nu$ be probability measures defined on a $\si$-algebra $\F$. Let $\mu_\G$ and $\nu_\G$ be the restrictions of $\mu$ and $\nu$ to a sub-$\si$-algebra $\G$ of $\F$.

Then \begin{equation*} D_{KL}(\mu_\G\pa\nu_\G)\le D_{KL}(\mu\pa\nu). \tag{1}\label{1} \end{equation*}

Indeed, \begin{equation*} D_{KL}(\mu\pa\nu)=\int d\nu\,h\Big(\frac{d\mu}{d\nu}\Big), \end{equation*} where $h(u):=u\ln u$ for $u\in(0,\infty)$, with $h(0):=0$ and $h(\infty)=\infty$. Next, letting $E_\nu(\cdot|\G)$ denote the conditional expectation given $\G$ w.r.t. $\nu$, for any nonnegative $\G$-measurable function $q$ we have \begin{equation*} \int d\nu_\G\,q\, E_\nu\Big(\frac{d\mu}{d\nu}\Big|\G\Big) = \int d\nu\,q\, E_\nu\Big(\frac{d\mu}{d\nu}\Big|\G\Big)\, =\int d\nu\,q\, \frac{d\mu}{d\nu}\, = \int d\mu\,q= \int d\mu_\G\,q; \end{equation*} the second equality in the latter display holds by the definition of the conditional expectation. So, \begin{equation*} \frac{d\mu_\G}{d\nu_\G}=E_\nu\Big(\frac{d\mu}{d\nu}\Big|\G\Big). \end{equation*} So, by Jensen's inequality applied to the convex function $h$, \begin{multline*} D_{KL}(\mu_\G\pa\nu_\G) =\int d\nu\,h\Big(\frac{d\mu_\G}{d\nu_\G}\Big) =\int d\nu\,h\Big(E_\nu\Big(\frac{d\mu}{d\nu}\Big|\G\Big)\Big) \\ \le \int d\nu\,E_\nu\Big( h\Big(\frac{d\mu}{d\nu}\Big)\Big|\G\Big) =\int d\nu\,h\Big(\frac{d\mu}{d\nu}\Big) =D_{KL}(\mu\pa\nu), \end{multline*} so that \eqref{1} follows.

Your second inequality now obtains by letting $\mu$ and $\nu$ be the distributions of $A$ and $B$, respectively, over the measurable space $(E_1,\F)$ and letting $\G$ be the sub-$\si$-algebra of $\F$ generated by the measurable map $f$. $\quad\Box$

$\endgroup$
2
$\begingroup$

The second inequality is typically called the "data-processing inequality". You can look up precise conditions on $f$ such that it is true (measurability suffices, but you may even assume it is a randomized transformation, typically called a "markov kernel" iirc).

The first inequality is not always true. Consider $A, B\sim\mathcal{N}(0,\sigma^2)$ independently. Let $f(x) = x+\mu$, and $g(x) = x$. Then $D_{KL}(A||B) = 0$, but $D_{KL}(f(A)||g(B)) \neq 0$, so the inequality cannot hold.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .