22
$\begingroup$

I'm looking for as many different arguments or derivations as possible that support the informal claim that Gaussian/Normal distributions are "the most fundamental" among all distributions.

A simple example: the central limit theorem (CLT) shows that the sum of i.i.d. random variables tends towards a Gaussian distribution.

$\endgroup$
16
  • 17
    $\begingroup$ The sum of iid square-integrable random variables tends toward Gaussian. People interested in heavy tails might disagree that this class is "most fundamental". $\endgroup$ Feb 2, 2021 at 5:23
  • 12
    $\begingroup$ Among all distributions with variance 1, it is the unique one that maximizes Shannon entropy. Among all distributions with Fisher information 1, it is the unique one that minimizes Shannon entropy. It is the unique distribution for which equality holds in the Cramer-Rao inequality. $\endgroup$
    – Deane Yang
    Feb 2, 2021 at 5:28
  • 8
    $\begingroup$ See also mathoverflow.net/q/40268/4600 $\endgroup$ Feb 2, 2021 at 5:57
  • 11
    $\begingroup$ For $n>1$, centered Gaussian measures are the only product probability measures on $\mathbf{R}^n$ which are rotation-invariant. $\endgroup$ Feb 2, 2021 at 7:45
  • 6
    $\begingroup$ See also stats.stackexchange.com/q/4364 $\endgroup$ Feb 2, 2021 at 10:44

7 Answers 7

22
$\begingroup$

The comments list many reasons why the Gaussian distribution is special, but is it "the most fundamental" among all distributions, as suggested in the OP? I would like to argue that (1) conservation laws are among the most fundamental laws of Nature, and (2) a quantity that obeys a conservation law will naturally follow an exponential -- rather than a Gaussian distribution.

Consider energy: two systems with energies $E_1$ and $E_2$ have total energy $E_1+E_2$, and if they are sufficiently large they will be independent, so the probability distribution must factorize: $P(E_1+E_2)=P(E_1)P(E_2)$, with the exponential distribution $P(E)\propto e^{-\beta E}$ as the unique normalizable solution (assuming $E$ is bounded from below).

This is the Gibbs measure. The Hammersley–Clifford theorem states that any probability measure which satisfies a Markov property is a Gibbs measure for an appropriate choice of (locally defined) energy function. The Gibbs measure is the fundamental measure of statistical physics, but it also applies widely outside of physics.

Economics is one such example. The statistical mechanics of money explains the exponential distribution of money as a direct consequence of the fact that money is conserved in general (there are exceptions). Wealth, in contrast, is not conserved (stocks may rise or fall), hence the non-exponential wealth distribution (Pareto distribution).

$\endgroup$
5
  • 3
    $\begingroup$ I'm familiar with the heuristic derivation of Gibbs measures, but I'm not sure I buy the argument that this tells us that exponential distributions are more natural... Measures aren't densities: why would the uniform density be the natural reference for energies? The natural reference measure is usually taken to be uniform on phase space, which then doesn't translate into uniform measure on energies. (But does translate into Gaussian measures when the energy is a quadratic function on phase space.)... $\endgroup$ Feb 2, 2021 at 16:20
  • 2
    $\begingroup$ ... The argument works to show that the Poisson process is natural, but then I can get a CLT from a Poisson process and not the other way around ;-) $\endgroup$ Feb 2, 2021 at 16:20
  • 5
    $\begingroup$ then again, the most natural choice of energy (as a function of phase space variables, aka cotangent bundle) is quadratic, $E\propto -p^2$, so Gaussian once again :-) (Equivalently, the natural measure for the PDF is $\sqrt E\mathrm dE$) $\endgroup$ Feb 2, 2021 at 20:24
  • 1
    $\begingroup$ (Actually, that's the number of states; the density is the differential $1/\sqrt E$.) $\endgroup$ Feb 2, 2021 at 21:11
  • $\begingroup$ I would beware of using any rule from the world of Economics (aka Voodoo Math) as an example $\endgroup$ Feb 3, 2021 at 14:14
19
$\begingroup$

If the random vector $(X,Y)$ in the plane has independent coordinates and a rotation-invariant distribution, then it is Gaussian.

$\endgroup$
3
  • 2
    $\begingroup$ Yes this is a really nice property of Gaussian distributions (and if I am not wrong then the proof follows by analyzing the characteristic function, I think). $\endgroup$ Feb 3, 2021 at 16:44
  • 1
    $\begingroup$ Motivated by this answer, I came across a fairly recent paper of Mike Christ which gives a quantitative version of this statement. It is here; arxiv.org/pdf/1506.00155.pdf $\endgroup$ Feb 10, 2021 at 17:09
  • 2
    $\begingroup$ Indeed, it's sufficient for $(X,Y)$ to be independent and invariant under rotation by the angle $\pi/4$ only. $\endgroup$ Feb 17, 2021 at 7:12
18
$\begingroup$

There is a whole book which addresses exactly this type of questions: suppose that a distribution has such and such properties, then it must be Gaussian (or sometimes Poisson).

MR0346969 Kagan, A. M.; Linnik, Yu. V.; Rao, C. Radhakrishna Characterization problems in mathematical statistics, John Wiley & Sons, New York-London-Sydney, 1973.

$\endgroup$
3
  • 1
    $\begingroup$ Chapter 7 of Probability Theory: The Logic of Science by E T Jaynes also discusses at length several reasons Gaussians are important, with an especial focus on why Gaussian errors are ubiquitous (so this isn't a CLT issue, because that's to do with sample means). $\endgroup$
    – J.G.
    Feb 3, 2021 at 16:19
  • $\begingroup$ @J.G.: I'm confused as to why you say the emergence of Gaussian errors is not related to CLT. The CLT measures the fluctuations from the mean, at the right scale where you can see such fluctuations. Intuitively I think of these fluctuations as "errors." $\endgroup$ Feb 3, 2021 at 16:57
  • 1
    $\begingroup$ @SamHopkins But the issue here is how one measurement's error is distributed. I discuss the chapter more fully here than a comment allows for. $\endgroup$
    – J.G.
    Feb 3, 2021 at 17:01
6
$\begingroup$

The Normal Distribution is the limit, in the sense of distributions, of the scaled sum of $n$ IID variables. This is the Central Limit Theorem.

I posted an outline of a proof of the Central Limit Theorem here:

In what follows, the fourier transform used is $$ \widehat{f}(\xi)=\int_{-\infty}^\infty f(x)\,e^{-i2\pi x\xi}\;\mathrm{d}x $$ Suppose we have a probability density function, $\phi$, which has a mean of $0$ and a variance of $1$ (this can be achieved by translation and scaling). Then, the following are true for the fourier transform of the pdf:

  1. $\widehat{\phi}(0)=1$ (i.e. $\int_{-\infty}^\infty\phi(x)\:\mathrm{d}x=1$; $\phi$ is a probability measure)

  2. $\widehat{\phi}\vphantom{\phi}^\prime(0)=0$ (i.e. $\int_{-\infty}^\infty (-i2\pi x)\phi(x)\:\mathrm{d}x=0$; the mean of $\phi$ is $0$)

  3. $\widehat{\phi}\vphantom{\phi}^{\prime\prime}(0)=-4\pi^2$ (i.e. $\int_{-\infty}^\infty (-i2\pi x)^2\phi(x)\:\mathrm{d}x=-4\pi^2$; the variance of the pdf is $1$)

Thus, to second order, the fourier transform of the pdf looks like $$ \widehat{\phi}(\xi)=1-2\pi^2\xi^2\tag{1} $$ The Central Limit Theorem looks at the sum of $k$ random variates contracted by $\sqrt{k}$ and scaled by $\sqrt{k}$. This maintains the unit integral while compensating for the increased variance due to summation.

The pdf of the sum of random variates is the convolution of the pdf of the variates. The fourier transform of a convolution is the product of the fourier transforms. The fourier transform of a function contracted by $\sqrt{k}$ and scaled by $\sqrt{k}$ is the fourier transform expanded by $\sqrt{k}$.

Thus, the fourier transform of the pdf of the sum of $k$ independent trials contracted and scaled appropriately is $$ \left(1-\frac{2\pi^2\xi^2}{k}\right)^k\tag{2} $$ which, for large $k$ approaches $$ e^{-2\pi^2\xi^2}\tag{3} $$ the inverse fourier transform of which is $$ \frac{1}{\sqrt{2\pi}}e^{-x^2/2}\tag{4} $$ The distribution in $(4)$ is gaussian normal distribution with mean $0$ and variance $1$.


The Normal Distribution is the probability distribution with mean $0$ and variance $\sigma^2$ that maximizes entropy.

I posted a derivation of the distribution which maximizes entropy here:

This answer gives an outline of how to use the Fourier Transform to prove that the $n$-fold convolution of any probability distribution with a finite variance contracted by a factor of $\sqrt{n}$ converges weakly to the normal distribution.

However, in his answer, Qiaochu Yuan mentions that one can use the Principle of Maximum Entropy to get a normal distribution. Below, I have endeavored to do just that using the Calculus of Variations.


Applying the Principle of Maximum Entropy

Suppose we want to maximize the entropy $$ -\int_{\mathbb{R}}\log(f(x))f(x)\,\mathrm{d}x\tag1 $$ over all $f$ whose mean is $0$ and variance is $\sigma^2$, that is $$ \int_{\mathbb{R}}\left(1,x,x^2\right)f(x)\,\mathrm{d}x=\left(1,0,\sigma^2\right)\tag2 $$ That is, we want the variation of $(1)$ to vanish $$ \int_{\mathbb{R}}(1+\log(f(x)))\,\delta f(x)\,\mathrm{d}x=0\tag3 $$ for all variations of $f$, $\delta f(x)$, so that the variation of $(2)$ vanishes $$ \int_{\mathbb{R}}\left(1,x,x^2\right)\delta f(x)\,\mathrm{d}x=(0,0,0)\tag4 $$ $(3)$, $(4)$, and orthogonality requires $$ \log(f(x))=c_0+c_1x+c_2x^2\tag5 $$ To satisfy $(2)$, we need $c_0=-\frac12\log\left(2\pi\sigma^2\right)$, $c_1=0$, and $c_2=-\frac1{2\sigma^2}$. That is, $$ \bbox[5px,border:2px solid #C0A000]{f(x)=\frac1{\sigma\sqrt{2\pi}}\,e^{-\frac{x^2}{2\sigma^2}}}\tag6 $$

$\endgroup$
3
$\begingroup$

Perhaps the reason lies in the characteristic function of the Gaussian distribution. It has several nice properties:

  1. As mentioned above by prof. Yuval Peres it has the rotational invariance property.

  2. Taking the Fourier transform of the normal distribution will again give you a normal distribution, and hence the central limit theorem. (This can also be viewed as a special consequence of the uncertainty principle.)

  3. It has finite moments.

  4. An $\mathbf{R}^n$ valued random variable $X = (X_1, ....., X_n)$ has the multivariate normal distribution if and only if any linear combination $a_1 X_1 + ..... + a_nX_n$ for deterministic real numbers $a_1 ,....... , a_n$ has the univariate normal distribution.

Although the 3rd point seems to be uninteresting in view of the several other distributions having this property, points 2 and 3 above are amongst the reasons why the Gaussian distribution is so common in nature.

$\endgroup$
3
  • 1
    $\begingroup$ Hi, could you elaborate a bit more on point 2? I don't understand why the fact that the normal distribution is a fixed point wrt Fourier transform implies CLT. $\endgroup$
    – LYH
    Feb 5, 2021 at 2:13
  • $\begingroup$ That follows from a direct computation ; take for instance $f(x) = e^{- \pi c x^2}$ for some $c>0$ and directly compute its Fourier transform to see what you get. $\endgroup$ Feb 6, 2021 at 17:30
  • $\begingroup$ @LYH Check the note at web.williams.edu/Mathematics/sjmiller/public_html/372Fa17/… $\endgroup$ Sep 28, 2021 at 9:07
2
$\begingroup$

When adding two independent random variables $X$ and $Y$, their respective moment generating functions $E\{\exp(t\,X))\}$ and $E\{\exp(t\,Y)\}$ are multiplied to yield the moment generating function of the sum $E\{\exp\bigl(t\,(X+Y)\bigr)\}$. This results in a convolution of the respective probability density functions.

If you take the logarithm of the moment generating functions (of the random variables and their sum), you get the cumulant generating functions that are added for the cumulant generating function of the sum of the independent variables.

Developing this function into a power series yields cumulants as successive coefficients.

The constant term is 0, the linear coefficient is the mean of the respective distributions (means of independent variables add when adding the variables), the quadratic coefficient is the variance, the next two terms are called skew and kurtosis. However, distributions for which all cumulants except mean and variance are zero are normal distributions.

The only cumulant guaranteed to be non-negative in any probability distribution is the variance: that is a significant contributor to the central limit theorem since variances cannot cancel each other.

$\endgroup$
0
$\begingroup$

If a Gaussian random process is stationary in the weak sense, then it is also stationary in the strict sense because the joint distribution of a (centred) Gaussian random vector is completely determined by the covariance matrix. This is not generally true for random processes with other distributions.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.