Mutual information in large deviation theory

Question

Many information theoretic quantities such as entropy and relative entropy appear in rate functions in large deviation theory (LDT). Is there any result in LDT that relates mutual information and rate function? I searched the internet and looked at many different books, but found no such results.

Answer 1

There's a few results. First of all there is the classical Sanov's Theorem.

One other result is about Gaussian measures.

For a centered Gaussian measure $\mu_0$ on Banach space $\mathcal B$ we can define the scaled measures $\mu_0^\epsilon$ by $\mu_0^\epsilon(A)=\mu_0(\epsilon^{-1}A)$. (For example if $\mu_0$ is the law of a continuous centered Gaussian process $X(t)$, as a Borel measure on the space of continuous functions, then $\mu_0^\epsilon$ is the law of $\epsilon X(t)$. )

The $\mu_0^\epsilon$ satisfy a LDP with rate function $FW(x)=\frac12\|x\|_{\mu_0}^2$ where $\|\cdot\|_{\mu_0}$ is the Cameron-Martin norm associated to $\mu_0$ (essentially it is $\|x\|_{\mu_0}=\|C^{-1/2}x\|$ where $C$ is the covariance operator).

It turns out that $FW(z)=D_{KL}(\mu^z||\mu_0)$ where $\mu^z$ is the shifted measure of $\mu_0$ by $z$.

Also see my (unanswered) question here What exactly is the relationship between Donsker-Varadhan variational formula and the Laplace principle?