This is a heuristic question that I think was once asked by Serge Lang. The gaussian: $e^{x^2}$ appears as the fixed point to the Fourier transform, in the punchline to the central limit theorem, as the solution to the heat equation, in a very nice proof of the AtiyahSinger index theorem etc. Is this an artifact of the techniques (such as the Fourier Transform) that people like use to deal with certain problems or is this the tip of some deeper platonic iceberg?

2$\begingroup$ I think the appearances you mention can be sensibly grouped in two broad classes: Fourieranalytic (including the CLT, see e.g. terrytao.wordpress.com/2010/01/05/…) and heatkernelish (incl. AtiyahSinger). Perhaps microlocal analysis is the best bridge between these. $\endgroup$ – Steve Huntsman Sep 28 '10 at 5:35

10$\begingroup$ I just want to point out that the appearance of the Gaussian in AtiyahSinger isn't so mysterious. In the McKeanSinger formula  $Index(D) = Tr_s (e^{tD^2})$  the Gaussian can be replaced with any smooth function which rapidly decays at infinity. The Gaussian is used because $e^{tD^2}$ is the solution operator for a very well understood differential equation, namely the heat equation, and thus we can look up the relevant asymptotic analysis in old PDE textbooks. $\endgroup$ – Paul Siegel Sep 28 '10 at 12:01

8$\begingroup$ There is a book "The Normal Distribution: Characterizations with Applications" by Bryc which could be taken as a large collection of probabilistic examples to add to your list. One of my favorites: the normal distribution in $\mathbb{R}^n$ is the unique (up to scaling) rotationinvariant probability measure with independent components. $\endgroup$ – Mark Meckes Sep 28 '10 at 12:11

5$\begingroup$ The central limit theorem and the heat equation make no mention of the gaussian in their basic setups; it turns out to be the answer to very natural questions. So I think those two (on the surface completely unrelated) examples show that this phenomenon is definitely not just an artifact of techniques. In particular, there are other methods of proving the CLT or solving the heat equation besides Fourier transforms. $\endgroup$ – Mark Meckes Sep 28 '10 at 13:01

3$\begingroup$ The factoring that appears in that volume calculation happens precisely because the standard normal distribution has independent components; the reason you can relate the normal distribution to the sphere is its rotation invariance. So that use of the Gaussian is an application of my favorite property which I mentioned above. As for the CLT and the heat equation, there certainly are deep relationships, but I don't know about "obvious" or "less heavy handed". $\endgroup$ – Mark Meckes Sep 28 '10 at 19:34
Quadratic (or bilinear) forms appear naturally throughout mathematics, for instance via inner product structures, or via dualisation of a linear transformation, or via Taylor expansion around the linearisation of a nonlinear operator. The LaplaceBeltrami operator and similar secondorder operators can be viewed as differential quadratic forms, for instance.
A Gaussian is basically the multiplicative or exponentiated version of a quadratic form, so it is quite natural that it comes up in multiplicative contexts, especially on spaces (such as Euclidean space) in which a natural bilinear or quadratic structure is already present.
Perhaps the one minor miracle, though, is that the Fourier transform of a Gaussian is again a Gaussian, although once one realises that the Fourier kernel is also an exponentiated bilinear form, this is not so surprising. But it does amplify the previous paragraph: thanks to Fourier duality, Gaussians not only come up in the context of spatial multiplication, but also frequency multiplication (e.g. convolutions, and hence CLT, or heat kernels).
One can also take an adelic viewpoint. When studying nonarchimedean fields such as the padics $Q_p$, compact subgroups such as $Z_p$ play a pivotal role. On the reals, it seems the natural analogue of these compact subgroups are the Gaussians (cf. Tate's thesis). One can sort of justify the existence and central role of Gaussians on the grounds that the real number system "needs" something like the compact subgroups that its nonarchimedean siblings enjoy, though this doesn't fully explain why Gaussians would then be exponentiated quadratic in nature.

1$\begingroup$ How exactly do you see a Gaussian as a subgroup of $\mathbb{R}$? $\endgroup$ – Chris Jones Aug 17 '17 at 16:32

1$\begingroup$ When you convolve Haar probability measure on a (compact) subgroup with itself, you get back the same measure, and this can in fact be used as a definition of such subgroups. If you convolve a Gaussian probability measure with itself, you almost get back the same Gaussian measure, but it has spread out by a factor of $\sqrt{2}$. So Gaussians are in some sense a "$\sqrt{2}$approximate group". $\endgroup$ – Terry Tao Aug 18 '17 at 1:43
I recently came across a strange and beautiful connection between the Gaussian $e^{x^2}$ and the method of least squares. It turns out that the square in $e^{x^2}$ and the square in ``least squares'' is the same square.
Let $(x_i,y_i)$ (with $1\leq i \leq n$) be the data set, and assume that for each $x$, the $y$'s are normally distributed with mean $\mu(x)=\alpha x+\beta$ and variance $\sigma^2$. Then, the likelihood of generating our data (assuming that the data points are independent) is $$\prod_{i=1}^n \frac{1}{\sqrt{2\pi \sigma^2}} \exp\left(\frac{(y_i\mu(x_i))^2}{2\sigma^2}\right) =\left(\frac{1}{\sqrt{2\pi \sigma^2}}\right)^n \exp\left( \frac{1}{2\sigma^2} \sum_{i=1}^n (y_i  \alpha x+\beta)^2 \right)$$ We would obviously want to choose the parameters $\alpha,\beta$ so that the likelihood is maximized, and this is accomplished by minimizing $$\sum_{i=1}^n (y_i  \alpha x+\beta)^2.$$ In other words, the least squares approximation is the one that makes the data set most likely to happen.

$\begingroup$ The analogous thing happens for the Laplace distribution and absolute loss, etc. There is certainly a relationship between loss criteria and distributions under which that loss minimizer is the MLE, but I'm not sure it goes any deeper. $\endgroup$ – R Hahn Sep 28 '10 at 19:35

1$\begingroup$ This is yet another manifestation of my favorite property, mentioned in the comments... $\endgroup$ – Mark Meckes Sep 28 '10 at 19:36

$\begingroup$ @R Hahn: See my comment below the question, (2nd from top). The relation does indeed go deeper. $\endgroup$ – Suvrit Sep 29 '10 at 13:30

1$\begingroup$ Isn't this connection the reason why the Gaussian is called Gaussian? If I remember correctly, Gauss introduced "least squares" as a regression technique not because its derivative gives rise to a linear problem but because the square fits the "normal error model", i.e. corresponds to the Gaussian distribution. $\endgroup$ – Dirk Nov 14 '12 at 12:51

2$\begingroup$ Sorry if this is a stupid question, but you assume that the data are normally distributed so why is it surprising that the PDF for the normal distribution come up? $\endgroup$ – user69208 Jun 24 '18 at 14:04
The sum of two independent normal random variables is again normal, i.e., the shape of the distribution is unchanged under addition except for stretching and scaling. Moreover, the normal distribution is unique among distributions with finite variance in having this property. Many phenomena in nature come from adding together various independent or almost independent terms. Therefore, we would expect the normal distribution to show up a lot in natureinspired mathematics.

12$\begingroup$ The normal distribution is not unique in this property. The same is true of all stable distributions. $\endgroup$ – Mark Meckes Sep 28 '10 at 12:06

1$\begingroup$ @Mark: Thanks, I have added the assumption of finite variance. $\endgroup$ – Bjørn KjosHanssen Sep 28 '10 at 17:12
This is just a minor amplification of one of Terry Tao's points. For any prime $p$, the ring $\mathbb{Z}_p$ of $p$adic integers forms an open compact additive subgroup of $\mathbb{Q}_p$, the completion of $\mathbb{Q}$ under the padic metric, and its characteristic function should be viewed as a padic analogue of the Gaussian. It displays many analogues of the nice properties we see from the Gaussian:
 It is smooth (in the sense that the smooth functions on totally disconnected spaces are defined as the locally constant functions, but this isn't completely tautological, since this class of functions turns out to be useful).
 It is taken to itself under the $p$adic Fourier transform when normalizations are chosen appropriately.
 It obeys something like a central limit theorem. For example, if you flip lots of coins, and ask for the number of heads mod $p^n$, you will, for sufficiently long trials, get a distribution that is close to uniform. It sounds like there could be a way to interpret this sort of convolution in terms of heat flowing, but I don't know a precise statement.
The situation with the real line is more complicated because it is connected but not compact, and therefore has no open compact subgroups. There is a maximal compact multiplicative monoid (occasionally called the "ring of integers of $\mathbb{R}$" informally), given by the closed interval $[1,1]$. You can think of the Gaussian as a smoothing of the uniform distribution on $[1,1]$, but it is not clear to me that this particular analogy is very fruitful.
I'm not an expert, but I believe Stein's method gives a more satisfying connection between the CLT and the heat equation, in particular one that does not involve the Fourier transform. Stein's characterization of the normal distribution and convergence to normality involves an operator closely related to the generator of the OrnsteinUhlenbeck process. On the other hand, the fact that latter has the Gaussian as its invariant measure can be obtained by a trivial transformation of the fundamental solution of the heat equation.
One of the reasons for the ubiquity of the Gaussian is displayed in what is probably the most electrifying half page of scientific prose ever writtenMaxwell's argument that the distribution of the velocities of molecules in the ideal gas is Gaussian (now known as the MaxwellBoltzmann distribution). The only physical assumptions used are that the density function depends only on the absolute value of the velocity (and not the direction) and that the components in the directions of the coordinate axes are statistically independent. Mathematically, ths means that the only functions in $3$space which depend only on the distance $r$ from the origin and which split as the product of three functions of one variable are those of the form $ae^{br^2}$. Maxwell does this by inspection but it is easy to give a rigorous proof (under very weak smoothness conditions) and the result holds, of course, in any dimension greater then or equal to $2$. Maxwell's reasoning can be found in his collected papers, or, more accesssibly, in Hawking's anthology "On the Shoulders of Giants".

2$\begingroup$ And this is yet another manifestation of the property I mentioned in the comments on the OP... $\endgroup$ – Mark Meckes Nov 14 '12 at 14:19