# Convexity and semicontinuity of the relative entropy function

There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.

Let $M$ be a closed manifold and $\mathcal{P}$ the set of Borel probability measures on $M$. Given a reference measure $\omega\in \mathcal{P}$ (usually the normalized Lebsgue measure), the relative entropy of a measure $\mu\ll\omega$ is defined as:

$$E(\mu|\omega)=\int_M\log\phi_\mu \ d\mu,$$

where $\displaystyle \phi_\mu=\frac{d\mu}{d\omega}$ is the Radon-Nykodim derivative (since we assume $\mu\ll\omega$). The integration is well defined since $\mu(\lbrace\phi_\mu=0\rbrace)=0$. For example $E(\mu|\omega)\ge0$ since

\begin{align*} E(\mu|\omega)&=\int\log\phi_\mu d\mu=\int_{\phi_\mu>0}-\log\frac{d\omega}{d\mu} d\mu \\\&\ge-\log\int_{\phi_\mu>0}\frac{d\omega}{d\mu} d\mu =-\log\omega(\lbrace\phi_\mu=0\rbrace)\ge0. \end{align*}

In particular $E(\mu|\omega)=0$ implies $\mu=\omega$.

Moreover, this function is convex: for all $\mu,\nu\ll\omega$, $$E(p\mu+q\nu|\omega)\le p\cdot E(\mu|\omega)+q\cdot E(\nu|\omega).$$

As noticed by Pablo (thank you!), the above claim is indeed a direct corollary of the convexity of $h(x)=x\log x$.

A more interesting statement I want to know if that, if $\mu_n\ll\omega$ such that $\mu_n\to\mu\not$$\ll\omega, will we always have E(\mu_n|\omega)\to+\infty? Thank you! Here \mu_n\to\mu in the sense that \mu_n(f)\to\mu(f) for all continuous functions f. The paper given by Ashok below provides an equivalent definition: \displaystyle E(\mu|\omega)=\sup_{\alpha}\sum_{A\in\alpha}\mu(A)\log\frac{\mu(A)}{\omega(A)}, where the supremum is taken over all finite, Borel partitions \alpha with \omega(A)>0. In particular if \mu\not$$\ll\omega$, we can take open sets $A_k$ with $\mu(A_k)\ge2\delta$ and $\omega(A_k)\to 0$. So $E(\mu|\omega)=+\infty$.

Now let's make a better choice of $A_k$'s such that $\mu(\partial A_k)=0$. Then if $\mu_n\to\mu$, $\mu_n(A_k)\to\mu(A_k)$ for all $k$. Hence we can pick $n_k$ such that $\mu_{n}(A_k)\ge\delta$ for all $n\ge n_k$. So for all $n\ge n_k$, we have $$E(\mu_{n}|\omega)\ge\mu_{n}(A_k)\log\frac{\mu_{n}(A_k)}{\omega(A_k)} \ge\delta\cdot\log\frac{\delta}{\omega(A_k)}\to\infty.$$

• Isn't this just the convexity of the (negative) entropy function $h(x)=x\log x$? If $\mu,\nu$ have densities $f,g$ (w.r.t. $\omega$), then $E(\mu|\omega)=\int \log f(x) f(x)d\omega(x)=\int h(f(x))d\omega(x)$ and likewise $E(\nu|\omega)=\int h(g(x))d\omega(x)$ and $E(p\mu+q\nu|\omega)=\int h(p f(x)+q g(x)) d\omega(x)$, so that the convexity should follow from the convexity of $h$ (I'm assuming that $p+q=1$). Apr 1, 2013 at 20:51
• Yes it really is! I will revise the question and ask for the another property. Apr 2, 2013 at 3:21
• Your second assertion is also true from the lower semicontinuity of $E(\mu|\omega)$ in $\mu$. Apr 2, 2013 at 4:17
• @ashok Could you say more about this? I don't know how to use the condition $\mu\not$$\ll\omega. Apr 2, 2013 at 6:03 • The second assertion is not true, one can get in the limit some odd measures such as c\cdot omega plus some delta measures say (think about putting some gaussians on your space multiplied by the usual Lebesgue measure), and the Radon-Nykodim derivative (which is defined by the Lebesgue decomposition in this case) will be equal to c a.e.. If by any case you are assuming that \mu,\omega are ergodic, then it is true, by the fact that they are different extreme points. @Ashok, usually the entropy is not contiuous wrt weak-* convergence, except for Yomdin or Newhouse theorems. – Asaf Apr 2, 2013 at 6:08 ## 2 Answers If M is complete and separable, then E(\mu|\omega) is lower semicontinuous in \mu on the set of all probability measures on M with respect to the weak convergence of probability measures, see Theorem 1 in section III of this paper. Once we have lower semicontinuity, we have$$ \liminf_{n\to \infty}E(\mu_n|\omega)\ge E(\mu|\omega)$$Since$\mu$is not absolutely continuous with respect to$\omega$, we have$E(\mu|\omega)=\infty$, hence the left hand side is also$\infty\$.

• The definition given there is easier to use. And the proof of lower semi-continuity is straightforward. Thank you! Apr 2, 2013 at 16:25

Here it is another proof: it works when $$\mu$$ and $$\omega$$ are finite measures.

It is straightforward to see that $$E(\mu | \omega ) - \mu (M) = \sup \left\{ \int_M f d \mu - \int_M e^{f} d \omega : f \in L^{\infty} ( \omega + \mu ) \right\}.$$

This is rather easy to prove: it relies on the fact that for $$a \geq0$$, we have $$at-e^t \leq a \log (a) - a$$ and the maximum is obtained when $$t=log(a)$$. Moreover, by a density argument we can prove that

$$E(\mu | \omega ) - \mu (M) = \sup \left\{ \int_M f d \mu - \int_M e^{f} d \omega : f \in C_b(M) \right\}.$$

Since the right hand side is the supremum of continuous functions in both $$\mu$$ and $$\omega$$ we can deduce that if $$\mu_n \rightharpoonup \mu$$ and $$\omega_n \rightharpoonup \omega$$ then $$\liminf_n E(\mu_n | \omega_n) \geq E(\mu | \omega),$$ that is, the relative entropy is jointly semicontinuous. Moreover we expressed the entropy as a supremum of linear functions in $$(\mu, \omega)$$ and so we have that it is convex in the couple $$(\mu, \omega)$$, that is $$E(t\mu + (1-t)\mu' | t\omega + (1-t)\omega' ) \leq tE(\mu|\omega) + (1-t) E(\mu'|\omega');$$ In particular it is convex in the first variable