There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.

Let $M$ be a closed manifold and $\mathcal{P}$ the set of Borel probability measures on $M$. Given a reference measure $\omega\in \mathcal{P}$ (usually the normalized Lebsgue measure), the relative entropy of a measure $\mu\ll\omega$ is defined as:

$$E(\mu|\omega)=\int_M\log\phi_\mu \ d\mu,$$

where $\displaystyle \phi_\mu=\frac{d\mu}{d\omega}$ is the Radon-Nykodim derivative (since we assume $\mu\ll\omega$). The integration is well defined since $\mu(\lbrace\phi_\mu=0\rbrace)=0$. For example $E(\mu|\omega)\ge0$ since

\begin{align*} E(\mu|\omega)&=\int\log\phi_\mu d\mu=\int_{\phi_\mu>0}-\log\frac{d\omega}{d\mu} d\mu \\\&\ge-\log\int_{\phi_\mu>0}\frac{d\omega}{d\mu} d\mu =-\log\omega(\lbrace\phi_\mu=0\rbrace)\ge0. \end{align*}

In particular $E(\mu|\omega)=0$ implies $\mu=\omega$.

Moreover, this function is convex: for all $\mu,\nu\ll\omega$, $$E(p\mu+q\nu|\omega)\le p\cdot E(\mu|\omega)+q\cdot E(\nu|\omega).$$

As noticed by Pablo (thank you!), the above claim is indeed a direct corollary of the convexity of $h(x)=x\log x$.

A more interesting statement I want to know if that, if $\mu_n\ll\omega$ such that $\mu_n\to\mu\not$$\ll\omega$, will we always have $E(\mu_n|\omega)\to+\infty$? Thank you!

Here $\mu_n\to\mu$ in the sense that $\mu_n(f)\to\mu(f)$ for all continuous functions $f$.

The paper given by Ashok below provides an equivalent definition: $\displaystyle E(\mu|\omega)=\sup_{\alpha}\sum_{A\in\alpha}\mu(A)\log\frac{\mu(A)}{\omega(A)}$, where the supremum is taken over all finite, Borel partitions $\alpha$ with $\omega(A)>0$.

In particular if $\mu\not$$\ll\omega$, we can take open sets $A_k$ with $\mu(A_k)\ge2\delta$ and $\omega(A_k)\to 0$. So $E(\mu|\omega)=+\infty$.

Now let's make a better choice of $A_k$'s such that $\mu(\partial A_k)=0$. Then if $\mu_n\to\mu$, $\mu_n(A_k)\to\mu(A_k)$ for all $k$. Hence we can pick $n_k$ such that $\mu_{n}(A_k)\ge\delta$ for all $n\ge n_k$. So for all $n\ge n_k$, we have $$E(\mu_{n}|\omega)\ge\mu_{n}(A_k)\log\frac{\mu_{n}(A_k)}{\omega(A_k)} \ge\delta\cdot\log\frac{\delta}{\omega(A_k)}\to\infty.$$

  • 2
    $\begingroup$ Isn't this just the convexity of the (negative) entropy function $h(x)=x\log x$? If $\mu,\nu$ have densities $f,g$ (w.r.t. $\omega$), then $E(\mu|\omega)=\int \log f(x) f(x)d\omega(x)=\int h(f(x))d\omega(x)$ and likewise $E(\nu|\omega)=\int h(g(x))d\omega(x)$ and $E(p\mu+q\nu|\omega)=\int h(p f(x)+q g(x)) d\omega(x)$, so that the convexity should follow from the convexity of $h$ (I'm assuming that $p+q=1$). $\endgroup$ Apr 1, 2013 at 20:51
  • $\begingroup$ Yes it really is! I will revise the question and ask for the another property. $\endgroup$
    – Pengfei
    Apr 2, 2013 at 3:21
  • $\begingroup$ Your second assertion is also true from the lower semicontinuity of $E(\mu|\omega)$ in $\mu$. $\endgroup$
    – Ashok
    Apr 2, 2013 at 4:17
  • $\begingroup$ @ashok Could you say more about this? I don't know how to use the condition $\mu\not$$\ll\omega$. $\endgroup$
    – Pengfei
    Apr 2, 2013 at 6:03
  • $\begingroup$ The second assertion is not true, one can get in the limit some odd measures such as $c\cdot omega$ plus some delta measures say (think about putting some gaussians on your space multiplied by the usual Lebesgue measure), and the Radon-Nykodim derivative (which is defined by the Lebesgue decomposition in this case) will be equal to $c$ a.e.. If by any case you are assuming that $\mu,\omega$ are ergodic, then it is true, by the fact that they are different extreme points. @Ashok, usually the entropy is not contiuous wrt weak-* convergence, except for Yomdin or Newhouse theorems. $\endgroup$
    – Asaf
    Apr 2, 2013 at 6:08

2 Answers 2


If $M$ is complete and separable, then $E(\mu|\omega)$ is lower semicontinuous in $\mu$ on the set of all probability measures on $M$ with respect to the weak convergence of probability measures, see Theorem 1 in section III of this paper.

Once we have lower semicontinuity, we have $$ \liminf_{n\to \infty}E(\mu_n|\omega)\ge E(\mu|\omega) $$

Since $\mu$ is not absolutely continuous with respect to $\omega$, we have $E(\mu|\omega)=\infty$, hence the left hand side is also $\infty$.

  • $\begingroup$ The definition given there is easier to use. And the proof of lower semi-continuity is straightforward. Thank you! $\endgroup$
    – Pengfei
    Apr 2, 2013 at 16:25

Here it is another proof: it works when $\mu$ and $\omega$ are finite measures.

It is straightforward to see that $$ E(\mu | \omega ) - \mu (M) = \sup \left\{ \int_M f d \mu - \int_M e^{f} d \omega : f \in L^{\infty} ( \omega + \mu ) \right\}. $$

This is rather easy to prove: it relies on the fact that for $a \geq0$, we have $at-e^t \leq a \log (a) - a$ and the maximum is obtained when $t=log(a)$. Moreover, by a density argument we can prove that

$$ E(\mu | \omega ) - \mu (M) = \sup \left\{ \int_M f d \mu - \int_M e^{f} d \omega : f \in C_b(M) \right\}. $$

Since the right hand side is the supremum of continuous functions in both $\mu$ and $\omega$ we can deduce that if $\mu_n \rightharpoonup \mu$ and $\omega_n \rightharpoonup \omega$ then $$ \liminf_n E(\mu_n | \omega_n) \geq E(\mu | \omega),$$ that is, the relative entropy is jointly semicontinuous. Moreover we expressed the entropy as a supremum of linear functions in $(\mu, \omega)$ and so we have that it is convex in the couple $(\mu, \omega)$, that is $$E(t\mu + (1-t)\mu' | t\omega + (1-t)\omega' ) \leq tE(\mu|\omega) + (1-t) E(\mu'|\omega');$$ In particular it is convex in the first variable


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.