# Convexity and semicontinuity of the relative entropy function

There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.

Let $M$ be a closed manifold and $\mathcal{P}$ the set of Borel probability measures on $M$. Given a reference measure $\omega\in \mathcal{P}$ (usually the normalized Lebsgue measure), the relative entropy of a measure $\mu\ll\omega$ is defined as:

$$E(\mu|\omega)=\int_M\log\phi_\mu \ d\mu,$$

where $\displaystyle \phi_\mu=\frac{d\mu}{d\omega}$ is the Radon-Nykodim derivative (since we assume $\mu\ll\omega$). The integration is well defined since $\mu(\lbrace\phi_\mu=0\rbrace)=0$. For example $E(\mu|\omega)\ge0$ since

\begin{align*} E(\mu|\omega)&=\int\log\phi_\mu d\mu=\int_{\phi_\mu>0}-\log\frac{d\omega}{d\mu} d\mu \\\&\ge-\log\int_{\phi_\mu>0}\frac{d\omega}{d\mu} d\mu =-\log\omega(\lbrace\phi_\mu=0\rbrace)\ge0. \end{align*}

In particular $E(\mu|\omega)=0$ implies $\mu=\omega$.

Moreover, this function is convex: for all $\mu,\nu\ll\omega$, $$E(p\mu+q\nu|\omega)\le p\cdot E(\mu|\omega)+q\cdot E(\nu|\omega).$$

As noticed by Pablo (thank you!), the above claim is indeed a direct corollary of the convexity of $h(x)=x\log x$.

A more interesting statement I want to know if that, if $\mu_n\ll\omega$ such that $\mu_n\to\mu\not$$\ll\omega, will we always have E(\mu_n|\omega)\to+\infty? Thank you! Here \mu_n\to\mu in the sense that \mu_n(f)\to\mu(f) for all continuous functions f. The paper given by Ashok below provides an equivalent definition: \displaystyle E(\mu|\omega)=\sup_{\alpha}\sum_{A\in\alpha}\mu(A)\log\frac{\mu(A)}{\omega(A)}, where the supremum is taken over all finite, Borel partitions \alpha with \omega(A)>0. In particular if \mu\not$$\ll\omega$, we can take open sets $A_k$ with $\mu(A_k)\ge2\delta$ and $\omega(A_k)\to 0$. So $E(\mu|\omega)=+\infty$.

Now let's make a better choice of $A_k$'s such that $\mu(\partial A_k)=0$. Then if $\mu_n\to\mu$, $\mu_n(A_k)\to\mu(A_k)$ for all $k$. Hence we can pick $n_k$ such that $\mu_{n}(A_k)\ge\delta$ for all $n\ge n_k$. So for all $n\ge n_k$, we have $$E(\mu_{n}|\omega)\ge\mu_{n}(A_k)\log\frac{\mu_{n}(A_k)}{\omega(A_k)} \ge\delta\cdot\log\frac{\delta}{\omega(A_k)}\to\infty.$$

-
Isn't this just the convexity of the (negative) entropy function $h(x)=x\log x$? If $\mu,\nu$ have densities $f,g$ (w.r.t. $\omega$), then $E(\mu|\omega)=\int \log f(x) f(x)d\omega(x)=\int h(f(x))d\omega(x)$ and likewise $E(\nu|\omega)=\int h(g(x))d\omega(x)$ and $E(p\mu+q\nu|\omega)=\int h(p f(x)+q g(x)) d\omega(x)$, so that the convexity should follow from the convexity of $h$ (I'm assuming that $p+q=1$). – Pablo Shmerkin Apr 1 '13 at 20:51
Yes it really is! I will revise the question and ask for the another property. – Pengfei Apr 2 '13 at 3:21
Your second assertion is also true from the lower semicontinuity of $E(\mu|\omega)$ in $\mu$. – Ashok Apr 2 '13 at 4:17

-