MathOverflow is a question and answer site for professional mathematicians. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

There are several different definitions of relative entropy, and some of them are not equivalent. Following is the definition we will use in this question.

Let $M$ be a closed manifold and $\mathcal{P}$ the set of Borel probability measures on $M$. Given a reference measure $\omega\in \mathcal{P}$ (usually the normalized Lebsgue measure), the relative entropy of a measure $\mu\ll\omega$ is defined as:

$$E(\mu|\omega)=\int_M\log\phi_\mu \ d\mu,$$

where $\displaystyle \phi_\mu=\frac{d\mu}{d\omega}$ is the Radon-Nykodim derivative (since we assume $\mu\ll\omega$). The integration is well defined since $\mu(\lbrace\phi_\mu=0\rbrace)=0$. For example $E(\mu|\omega)\ge0$ since

\begin{align*} E(\mu|\omega)&=\int\log\phi_\mu d\mu=\int_{\phi_\mu>0}-\log\frac{d\omega}{d\mu} d\mu \\\&\ge-\log\int_{\phi_\mu>0}\frac{d\omega}{d\mu} d\mu =-\log\omega(\lbrace\phi_\mu=0\rbrace)\ge0. \end{align*}

In particular $E(\mu|\omega)=0$ implies $\mu=\omega$.

Moreover, this function is convex: for all $\mu,\nu\ll\omega$, $$E(p\mu+q\nu|\omega)\le p\cdot E(\mu|\omega)+q\cdot E(\nu|\omega).$$

As noticed by Pablo (thank you!), the above claim is indeed a direct corollary of the convexity of $h(x)=x\log x$.

A more interesting statement I want to know if that, if $\mu_n\ll\omega$ such that $\mu_n\to\mu\not$$\ll\omega$, will we always have $E(\mu_n|\omega)\to+\infty$? Thank you!

Here $\mu_n\to\mu$ in the sense that $\mu_n(f)\to\mu(f)$ for all continuous functions $f$.

The paper given by Ashok below provides an equivalent definition: $\displaystyle E(\mu|\omega)=\sup_{\alpha}\sum_{A\in\alpha}\mu(A)\log\frac{\mu(A)}{\omega(A)}$, where the supremum is taken over all finite, Borel partitions $\alpha$ with $\omega(A)>0$.

In particular if $\mu\not$$\ll\omega$, we can take open sets $A_k$ with $\mu(A_k)\ge2\delta$ and $\omega(A_k)\to 0$. So $E(\mu|\omega)=+\infty$.

Now let's make a better choice of $A_k$'s such that $\mu(\partial A_k)=0$. Then if $\mu_n\to\mu$, $\mu_n(A_k)\to\mu(A_k)$ for all $k$. Hence we can pick $n_k$ such that $\mu_{n}(A_k)\ge\delta$ for all $n\ge n_k$. So for all $n\ge n_k$, we have $$E(\mu_{n}|\omega)\ge\mu_{n}(A_k)\log\frac{\mu_{n}(A_k)}{\omega(A_k)} \ge\delta\cdot\log\frac{\delta}{\omega(A_k)}\to\infty.$$

share|cite|improve this question
Isn't this just the convexity of the (negative) entropy function $h(x)=x\log x$? If $\mu,\nu$ have densities $f,g$ (w.r.t. $\omega$), then $E(\mu|\omega)=\int \log f(x) f(x)d\omega(x)=\int h(f(x))d\omega(x)$ and likewise $E(\nu|\omega)=\int h(g(x))d\omega(x)$ and $E(p\mu+q\nu|\omega)=\int h(p f(x)+q g(x)) d\omega(x)$, so that the convexity should follow from the convexity of $h$ (I'm assuming that $p+q=1$). – Pablo Shmerkin Apr 1 '13 at 20:51
Yes it really is! I will revise the question and ask for the another property. – Pengfei Apr 2 '13 at 3:21
Your second assertion is also true from the lower semicontinuity of $E(\mu|\omega)$ in $\mu$. – Ashok Apr 2 '13 at 4:17
@ashok Could you say more about this? I don't know how to use the condition $\mu\not$$\ll\omega$. – Pengfei Apr 2 '13 at 6:03
The second assertion is not true, one can get in the limit some odd measures such as $c\cdot omega$ plus some delta measures say (think about putting some gaussians on your space multiplied by the usual Lebesgue measure), and the Radon-Nykodim derivative (which is defined by the Lebesgue decomposition in this case) will be equal to $c$ a.e.. If by any case you are assuming that $\mu,\omega$ are ergodic, then it is true, by the fact that they are different extreme points. @Ashok, usually the entropy is not contiuous wrt weak-* convergence, except for Yomdin or Newhouse theorems. – Asaf Apr 2 '13 at 6:08
up vote 1 down vote accepted

If $M$ is complete and separable, then $E(\mu|\omega)$ is lower semicontinuous in $\mu$ on the set of all probability measures on $M$ with respect to the weak convergence of probability measures, see Theorem 1 in section III of this paper.

Once we have lower semicontinuity, we have $$ \liminf_{n\to \infty}E(\mu_n|\omega)\ge E(\mu|\omega) $$

Since $\mu$ is not absolutely continuous with respect to $\omega$, we have $E(\mu|\omega)=\infty$, hence the left hand side is also $\infty$.

share|cite|improve this answer
The definition given there is easier to use. And the proof of lower semi-continuity is straightforward. Thank you! – Pengfei Apr 2 '13 at 16:25

Here it is another proof: it works when $\mu$ and $\omega$ are finite measures.

It is straightforward to see that $$ E(\mu | \omega ) + \mu (M) = \sup \left\{ \int_M f d \mu - \int_M e^{f} d \omega : f \in L^{\infty} ( \omega + \mu ) \right\}. $$

This is rather easy to prove: it relies on the fact that for $a \geq0$, we have $at-e^t \leq a \log (a) - a$ and the maximum is obtained when $t=log(a)$. Moreover, by a density argument we can prove that

$$ E(\mu | \omega ) + \mu (M) = \sup \left\{ \int_M f d \mu - \int_M e^{f} d \omega : f \in C_b(M) \right\}. $$

Since the right hand side is the supremum of continuous functions in both $\mu$ and $\nu$ we can deduce that if $\mu_n \rightharpoonup \mu$ and $\omega_n \rightharpoonup \omega$ then $$ \liminf_n E(\mu_n | \omega_n) \geq E(\mu | \omega),$$ that is, the relative entropy is jointly semicontinuous. Moreover we expressed the entropy as a supremum of linear functions in $(\mu, \omega)$ and so we have that it is convex in the couple $(\mu, \omega)$, that is $$E(t\mu + (1-t)\mu' | t\omega + (1-t)\omega' ) \leq tE(\mu|\omega) + (1-t) E(\mu'|\omega');$$ In particular it is convex in the first variable

share|cite|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.