Gaussian distribution, maximum entropy and the heat equation

Question

I have asked this question on MathSE, but I got no replies, so I thought of trying here.

---

Consider the Gaussian distribution on $\mathbb{R}$ with mean $m$ and variance $t=\sigma^2$. This has the explicit density

$$ f(x,m,t) = \frac{1}{\sqrt{2\pi t}}e^{-\frac{(x-m)^2}{2t}} $$

The Gaussian distribution has many characterizations, and two of them are the following:

1. The Gaussian distribution maximizes entropy amongst all the distributions on $\mathbb{R}$ with mean $m$ and variance $t$.

2. The density $f(x,m,t)$ provides a solution of the heat equation. More precisely, consider the differential equation $$ \begin{cases} \frac{\partial u}{\partial t} &= \frac{\partial^2 u }{\partial x^2} \quad \text{ for } x\in \mathbb{R},t>0 \\ u(x,0) &= \delta_m(x) \end{cases} $$ where $\delta_m$ is the Dirac concentrated at $m$. Then the function $u(x,t)=f(x,m,2t)$ is a solution of this differential equation.

I would like to know whether there is a reason (intuitive or more formal) why the solution of the two problems (maximum entropy distribution and heat equation) is the same. Of course, an explanation is that we can solve explicitly both problems and the solution happens to be the same, but I wonder whether there is a more conceptual reason for this.

Answer 1

Both the Gaussian maximum entropy distribution and the Gaussian solution of the diffusion equation (heat equation) follow from the central limit theorem, that the limiting distribution of the sum of i.i.d. random variables with given average and variance is a Gaussian. The connection between the central limit theorem and the diffusion equation, which describes a random walk with zero mean and unit variance, is obvious; the connection between the central limit theorem and the maximum entropy distribution is less obvious. For pointers to the literature on the latter connection, see these MO question and answers.

Answer 2

Besides the central limit theorem, there is the connection between diffusion and Wasserstein distance $W_2(p,q)$ (the minimum integral of squared distance from $x$ to $T(x)$ when $T$ maps $p$ to $q$) : for that metric the heat equation is the gradient flow of $\Phi(p)=-\int p\log p$.

http://www-dimat.unipv.it/savare/Ravello2010/ravelloC.pdf

Answer 3

Here is yet another late answer, but I hope it is relevant. Let me make first make clear that I use the mathematical "minus entropy" convention (as is common in my field, which is optimal transport). In other words, for me the entropy of a probability distribution $u$ is $$ \mathcal H(u)=+\int_{\mathbb R}u \log u. $$ A beautiful result due to R. Jordan, D. Kinderlehrer, and F. Otto tells us that one can interpret the heat equation $\partial_t u=\Delta u$ as the gradient-flow of the entropy, $$ \partial_t u=-\operatorname{grad}_{W_2} \mathcal H(u) $$ with respect to the quadratic wasserstein metric $W_2$ over $\mathbb R$ (or $\mathbb R^d$, for that matters). Regardless of the interpretation of this Wasserstein gradient (which would deserve a lot more than this simple post, see e.g. C. Villani's books), this tells us that the entropy (or, again, minus the entropy depending on one's conveniton) decreases (increases) as fast as possible and tends to be minimized (maximized) along the time-evolution.

Now, from the purely PDE perspective it is easy to check that the evolution preserves the mass and the average, and that the variance grows linearly in time, which means here that $$ \int u(t,x)dx=1, \qquad \int xu(t,x) dx=m, \qquad \int x^2 u(t,x)=t $$ since we started from $u(0,.)=\delta_m(.)$. (Simply differentiate w.r.t time under the integral signs, use the PDE to substitute $\Delta u$ for $\partial_t u$, and finally integrate by parts.)

Owing to the very gradient flow structure (entropy tends to diminish as fast as possible) it becomes then very likely (and true indeed!) that the solution $u(t,.)$ of the heat flow at time $t$ is minimizing the entropy among all probability distributions satisfying the constraints, i-e average $m$ and variance $t$. OF course that's not a proof, but still a damn good hint I believe.