Convexity and semicontinuity of the relative entropy function

Question

There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.

Let $M$ be a closed manifold and $\mathcal{P}$ the set of Borel probability measures on $M$. Given a reference measure $\omega\in \mathcal{P}$ (usually the normalized Lebsgue measure), the relative entropy of a measure $\mu\ll\omega$ is defined as:

$$E(\mu|\omega)=\int_M\log\phi_\mu \ d\mu,$$

where $\displaystyle \phi_\mu=\frac{d\mu}{d\omega}$ is the Radon-Nykodim derivative (since we assume $\mu\ll\omega$). The integration is well defined since $\mu(\lbrace\phi_\mu=0\rbrace)=0$. For example $E(\mu|\omega)\ge0$ since

\begin{align*} E(\mu|\omega)&=\int\log\phi_\mu d\mu=\int_{\phi_\mu>0}-\log\frac{d\omega}{d\mu} d\mu \\\&\ge-\log\int_{\phi_\mu>0}\frac{d\omega}{d\mu} d\mu =-\log\omega(\lbrace\phi_\mu=0\rbrace)\ge0. \end{align*}

In particular $E(\mu|\omega)=0$ implies $\mu=\omega$.

Moreover, this function is convex: for all $\mu,\nu\ll\omega$, $$E(p\mu+q\nu|\omega)\le p\cdot E(\mu|\omega)+q\cdot E(\nu|\omega).$$

As noticed by Pablo (thank you!), the above claim is indeed a direct corollary of the convexity of $h(x)=x\log x$.

---

A more interesting statement I want to know if that, if $\mu_n\ll\omega$ such that $\mu_n\to\mu\not$$\ll\omega$, will we always have $E(\mu_n|\omega)\to+\infty$? Thank you!

Here $\mu_n\to\mu$ in the sense that $\mu_n(f)\to\mu(f)$ for all continuous functions $f$.

---

The paper given by Ashok below provides an equivalent definition: $\displaystyle E(\mu|\omega)=\sup_{\alpha}\sum_{A\in\alpha}\mu(A)\log\frac{\mu(A)}{\omega(A)}$, where the supremum is taken over all finite, Borel partitions $\alpha$ with $\omega(A)>0$.

In particular if $\mu\not$$\ll\omega$, we can take open sets $A_k$ with $\mu(A_k)\ge2\delta$ and $\omega(A_k)\to 0$. So $E(\mu|\omega)=+\infty$.

Now let's make a better choice of $A_k$'s such that $\mu(\partial A_k)=0$. Then if $\mu_n\to\mu$, $\mu_n(A_k)\to\mu(A_k)$ for all $k$. Hence we can pick $n_k$ such that $\mu_{n}(A_k)\ge\delta$ for all $n\ge n_k$. So for all $n\ge n_k$, we have $$E(\mu_{n}|\omega)\ge\mu_{n}(A_k)\log\frac{\mu_{n}(A_k)}{\omega(A_k)} \ge\delta\cdot\log\frac{\delta}{\omega(A_k)}\to\infty.$$

Answer 1

If $M$ is complete and separable, then $E(\mu|\omega)$ is lower semicontinuous in $\mu$ on the set of all probability measures on $M$ with respect to the weak convergence of probability measures, see Theorem 1 in section III of this paper.

Once we have lower semicontinuity, we have $$ \liminf_{n\to \infty}E(\mu_n|\omega)\ge E(\mu|\omega) $$

Since $\mu$ is not absolutely continuous with respect to $\omega$, we have $E(\mu|\omega)=\infty$, hence the left hand side is also $\infty$.

Answer 2

Here it is another proof: it works when $\mu$ and $\omega$ are finite measures.

It is straightforward to see that $$ E(\mu | \omega ) - \mu (M) = \sup \left\{ \int_M f d \mu - \int_M e^{f} d \omega : f \in L^{\infty} ( \omega + \mu ) \right\}. $$

This is rather easy to prove: it relies on the fact that for $a \geq0$, we have $at-e^t \leq a \log (a) - a$ and the maximum is obtained when $t=log(a)$. Moreover, by a density argument we can prove that

$$ E(\mu | \omega ) - \mu (M) = \sup \left\{ \int_M f d \mu - \int_M e^{f} d \omega : f \in C_b(M) \right\}. $$

Since the right hand side is the supremum of continuous functions in both $\mu$ and $\omega$ we can deduce that if $\mu_n \rightharpoonup \mu$ and $\omega_n \rightharpoonup \omega$ then $$ \liminf_n E(\mu_n | \omega_n) \geq E(\mu | \omega),$$ that is, the relative entropy is jointly semicontinuous. Moreover we expressed the entropy as a supremum of linear functions in $(\mu, \omega)$ and so we have that it is convex in the couple $(\mu, \omega)$, that is $$E(t\mu + (1-t)\mu' | t\omega + (1-t)\omega' ) \leq tE(\mu|\omega) + (1-t) E(\mu'|\omega');$$ In particular it is convex in the first variable