3
$\begingroup$

Many information theoretic quantities such as entropy and relative entropy appear in rate functions in large deviation theory (LDT). Is there any result in LDT that relates mutual information and rate function? I searched the internet and looked at many different books, but found no such results.

$\endgroup$

1 Answer 1

3
$\begingroup$

There's a few results. First of all there is the classical Sanov's Theorem.

One other result is about Gaussian measures.

For a centered Gaussian measure $\mu_0$ on Banach space $\mathcal B$ we can define the scaled measures $\mu_0^\epsilon$ by $\mu_0^\epsilon(A)=\mu_0(\epsilon^{-1}A)$. (For example if $\mu_0$ is the law of a continuous centered Gaussian process $X(t)$, as a Borel measure on the space of continuous functions, then $\mu_0^\epsilon$ is the law of $\epsilon X(t)$. )

The $\mu_0^\epsilon$ satisfy a LDP with rate function $FW(x)=\frac12\|x\|_{\mu_0}^2$ where $\|\cdot\|_{\mu_0}$ is the Cameron-Martin norm associated to $\mu_0$ (essentially it is $\|x\|_{\mu_0}=\|C^{-1/2}x\|$ where $C$ is the covariance operator).

It turns out that $FW(z)=D_{KL}(\mu^z||\mu_0)$ where $\mu^z$ is the shifted measure of $\mu_0$ by $z$.

Also see my (unanswered) question here What exactly is the relationship between Donsker-Varadhan variational formula and the Laplace principle?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.