Central limit theorem via maximal entropy

Question

Let $\rho(x)$ be a probability density function on $\mathbb{R}$ with prescribed variance $\sigma^2$, so that: $$\int_\mathbb{R} \rho(x)\, dx = 1$$ and $$\int_\mathbb{R} x^2 \rho(x), dx = \sigma^2$$ Fact: the density function which maximizes the entropy functional $$S(\rho) = -\int_\mathbb{R} \rho(x) \log \rho(x)\, dx$$ with the constraints above is the normal distribution $$\rho(x) = \frac{1}{\sqrt{2\pi \sigma^2}} e^{-\frac{x^2}{2\sigma^2}}$$ This can be proved using basic techniques from the calculus of variations.

My question: can this be used to prove the central limit theorem? In other words, can one show directly that the limiting distribution of the average of a sequence of i.i.d. random variables maximizes entropy?

Actually, I don't care too much about entropy. I'm mainly interested in the possibility of a variational proof of the central limit theorem.

Answer 1

There's a 1985 article by Derriennic called "Entropie, theoremes limite et marches aleatoires" (entropy, limit theorems and random walks). In it there is a section where the connection between your observation that the Gaussian maximizes entropy (which is attributed to Shannon) and the central limit theorem is discussed.

He begins by discussing a proof (attributed to Pinsker, citing page 20 of his book on Information Stability) that the iterated convolution of a density on a compact group converges to a constant. The proof is based on the fact that the entropy of the sequence of convolutions is monotone.

After this he discusses a work of Csizar (A note on limitimg distributions on topological groups) where the same technique is used to prove the convergence of convolutions of general probabilities on a compact group (not supported on a closed subgroup) to the Haar measure.

Finally, answering your question, the proof of the central limit theorem in $\mathbb{R}$ using the idea of entropy monotonicity is attributed to Linnik.

The precise reference being: "An information-theoretic proof of the central limit theorem with the Lindeberg condition", Theory of Probability and its applications. 1959, Vol IV, n o 3, 288-299.

Answer 2

There is a book on the subject: "Information Theory and The Central Limit Theorem" by Oliver Johnson. The article by Anshelevich mentioned by Yemon considers the operator $T$ acting on probability densities and corresponding to going from the law of a random variable $X$ to that of $(X+Y)/\sqrt{2}$ where $Y$ is an independent copy of $X$. The entropy is a Lyapunov function for this transformation which is the simplest example of a renormalization group transformation. The $N(0,1)$ is a fixed point and it is easy to diagonalize the linearization of $T$ near this fixed point using Wick monomials, i.e., Hermite polynomials. The directions corresponding to 0-th, 1-st and 2-nd moments are expanding (relevant operators) or neutral (marginal operators) while all others are contracting (irrelevant operators). Therefore if one makes the necessary arrangements (renormalization conditions) to fix these moments (e.g. subtracting $N$ times the mean and dividing by $\sqrt{N}$) then one lies on the stable manifold of the Gaussian fixed point. See the textbook on probability theory by Koralov and Sinai for more details. The generalization of the $T$ map for joint probability distributions of dependent variables, i.e., the renormalization group is explained in the book "A Renormalization Group Analysis of the Hierarchical Model in Statistical Mechanics" by Collet and Eckmann. The issue with using this type of nonlinear transformations is that the above diagonalization at a fixed point only gives information about the vicinity of that fixed point. To get results far away, having a Lyapunov function like the entropy is of great importance. This is an active area in physics which investigates generalizations of Zamolodchikov's $c$-"theorem" in conformal field theory. See for instance this article for a recent review. Entanglement entropy seems to be the Lyapunov function in this setting.

Answer 3

With some more assumptions, the answer is yes.

There is one theorem called the entropy power inequality (EPI). It says the entropy power of the sum of two independent random variables is no less than the sum of their entropy powers, i.e. $2^{2h(X+Y)}\ge 2^{2h(X)}+2^{2h(Y)}$. Since Gaussian maximized entropy under 2nd moment constraint, this theorem basically says the sum of two independent random variable is distributed more like a Gaussian.

The proof of course includes more details and techniques. You can show, for example, that the KL divergence between a properly normalized sum of triangular array and a Gaussian distribution converges to 0, which implied weak convergence by Pinsker's inequality.

Answer 4

See recent article "Entropy and the discrete central limit theorem". We say that $Y$ has a lattice distribution with span $h>0$ if its support is a subset of $\{a+kh\;:\;k\in\mathbb N\}$ for some $a\in\mathbb R$; the span $h$ is maximal if it is the largest such $h$. We write $D(Y)$ for the relative entropy $D(p\|q)$ between the probability mass function $p$ of $Y$ and the probability mass function $q$ of a Gaussian random variable $Z\sim N(\mu,\sigma^2)$ quantised on $A$ as, $$q(a+kh)=\int_{a+kh}^{a+(k+1)h}\phi(x)dx,\qquad k\in\mathbb N,$$

(Discrete entropic CLT)If $\hat{S}_n$, $n\geq 1$, are the standardised sums of a sequence $\{X_n\}$ of i.i.d. lattice random variables with finite variance, then: $$D\Big(\frac{1}{\sqrt{n}}\sum_{i=1}^n(X_i-\mu)\Big) =D(\hat{S}_n)\to 0,\qquad\mbox{as}\;n\to\infty. $$

Pinsker's inequality combined with the triangle inequality for the total variation norm imply a strong version of the CLT: Taking $\mu=0$ without loss of generality, let $Z\sim N(0,\sigma^2)$ and let $Z_n$ be the quantised Gaussian as in the definition of $D(\hat{S}_n)$. Then, $$\|\hat{S}_n-Z\|_{\rm TV}\leq \sqrt{\frac{1}{2}D(\hat{S}_n)}+\|Z_n-Z\|_{\rm TV}\to 0, \qquad \mbox{as}\;n\to\infty,$$ and this implies local-CLT-like results.