When we want to find the standard deviation of $\{1,2,2,3,5\}$ we do
$$\sigma = \sqrt{ {1 \over 5-1} \left( (1-2.6)^2 + (2-2.6)^2 + (2-2.6)^2 + (3-2.6)^2 + (5 - 2.6)^2 \right) } \approx 1.52$$.
Why do we need to square and then square-root the numbers?
One answer is mathematical convenience. The theory is much simpler using powers of two rather than other powers.
There are justifications. Squaring makes small numbers smaller and makes big numbers bigger. You could argue that a useful measure of dispersion should be forgiving of small errors but weigh larger errors more heavily, so it makes sense to square the deviations from the mean.
Intro by Reid Barton
I think the answer should involve the additivity of variance for independent variables and the central limit theorem. Maybe someone can flesh this out.
Answer
Indeed, the variance has the additive property: if $r_1$ and $r_2$ are random variables with means $\mu_1, \mu_2$ and variances $d_1, d_2$, and these two variables are independent, then the new random variable $r = r_1+r_2$ has the mean $\mu_1+\mu_2$ and variance $d_1+d_2$.
Moreover, suppose we sum a large number $N$ of independent copies of our random variable $r$ with mean $\mu$ and variance $d$. Under mild assumptions, the central limit says the distribution will approach a normal distribution, which by the above has mean $N\mu$ and variance $Nd$. Observe that a normal distribution is completely determined by its mean and variance. We conclude that the only parameters of a distribution that we can observe from the sum of many independent copies of the distribution are the mean and variance.
Now that we established how good it is to square numbers, to get variance, the standard deviation has a very easy explanation: it's the only way to get back from variance to something with the dimension of our original set. That is, suppose you numbers are some lengths written in meters. Since the variance is meters squared, you have to take the square root to get something that can be compared with the original set.
Now, honestly, this not the only way, since you could also, e.g., multiply it by 2. That's why it's called standard deviation — to show that among different numerical constants we've chosen a specific one.
Short answer: You could argue that the most natural thing to do when defining a "standard deviation-type" quantity is to use an absolute value: $E(|X|)$, but it's really annoying to deal with absolute values under the expectation, so we use the next best thing: $\sqrt{E( X^2 )}$. You still get something positive and it's easier to deal with the square inside. We take a square root at the end to get something with the same "units" as $X$.
Long answer: It's often helpful to think of random variables as living in the function space $L^2(\Omega)$, and in this setting, this computation gives the $L^2$ norm of the centered random variable $X - EX$. Also, with this perspective, the covariance defines an inner product.
L^2 observation makes a lot of sense and is quite deep.
$\endgroup$
Commented
Oct 24, 2009 at 20:31
If you apply Bessel's correction — dividing by $5-1$ rather than by $5$ when you have $5$ numbers — then some of the otherwise right things stated in some of the answers are wrong. Bessel's correction is intended to be used only when the variance one is computing is based on a sample to be used to estimate the variance of the whole population.
I won't be surprised if nobody used the variance and standard deviation before Abraham de Moivre did so in the 18th century. De Moivre considered this question: If you toss a fair coin $1800$ times, what is the probability that the number of heads is in some specified range? You have a binomial distribution, and computing its exact values was not feasible. De Moivre approximated the distribution of the number of heads with a normal distribution with the same mean and the same standard deviation. In so doing, he was the first to introduce the normal distribution, and the first to prove a special case of the central limit theorem. The normal distribution with mean $0$ and variance $1$ is $$ \varphi(x)\,dx=\frac 1 {\sqrt{2\pi}} e^{-x^2/2}\,dx $$ and with mean $\mu$ and variance $\sigma^2$ it is $$ \varphi\left(\frac{x-\mu}\sigma\right)\, \frac{dx}\sigma. $$ It's easy to find the mean and standard deviation for the number of heads when one fair coin is tossed: they're both $1/2$. How do you do it for the sum of $1800$ independent copies of that random variable? De Moivre found that the mean-square deviation is additive: for independent random variables $X_1,\dotsc,X_{1800}$ one has $\operatorname{var}(X_1+\dotsb+X_{1800})=\operatorname{var}(X_1)+\dotsb+\operatorname{var}(X_{1800})$. You cannot do that with mean absolute deviation. If I'm recalling some details correctly, he published these findings in a paper in Latin while he lived in France, and at that time he gave the normal distribution as $$ C e^{-x^2/2}\,dx $$ where he could find $C$ only numerically. Later he went to England to escape the persecution of Protestants and met James Stirling, who showed that $C=1/\sqrt{2\pi}$. De Moivre wrote a book in English called The Doctrine of Chances, which I think was 18th-century English for the theory of probability. Some have speculated that the Reverend Thomas Bayes may have studied under him, but I don't know that that's gone beyond speculation.
(If you want to know the probability that the number of heads is $\ge894$, note that that's the same as “${>893}$”, and find the probability that the normally distributed random variable with the same mean and variance is $>893.5$. That is a "continuity correction" and works surprisingly well even for fairly small samples.)
On to Bessel's correction: When does one use $$ \frac{(x_1-\bar x)^2+\cdots+(x_n-\bar x)^2}{n-1}, $$ where $\bar x=(x_1+\cdots+x_n)/n$, with $n-1$ rather than $n$ in the denominator? As you can see from simple examples, that will not serve de Moivre's purpose described above: it's not additive.
If $X_1,\dotsc,X_n$ are an independent sample from a population with mean $\mu$ and variance $\sigma^2$, then the expected value of $$ \frac{(X_1-\mu)^2+\dotsb+(X_n-\mu)^2} n \tag 1\label{1} $$ is $\sigma^2$. But if one has only the sample and not the whole population, one doesn't know $\mu$ and one can use the sample mean $\bar X$ as an estimate of $\mu$. But the expected value of $$ \frac{(X_1-\bar X)^2+\dotsb+(X_n-\bar X)^2} n $$ is smaller than the expected value of \eqref{1}. Specifically, a bit of algebra shows that $$ \sum_{i=1}^n (X_i-\mu)^2 = \left( \sum_{i=1}^n (X_i-\bar X)^2 \right) + n(\bar X-\mu)^2, \tag 2\label{2} $$ and since the expectation of the last term is $\sigma^2$, that of the first term on the right in \eqref{2} must be $(n-1)\sigma^2$. Thus Bessel's correction gives an unbiased estimate of the population variance $\sigma^2$. (But its square root does not give an unbiased estimate of the population standard deviation. And unbiasedness is at best somewhat overrated anyway, and in some cases is a very very bad thing (I had a paper in the American Mathematical Monthly a few years ago demonstrating how bad it can sometimes be).
One answer I've heard is that you want the notion of standard deviation to 1) apply to points in Euclidean space, and 2) to be invariant under rotation. You don't get the second property unless you square the distances.
The answers here that stress that "the answers here that stress convenience are missing the crucial point" are missing a crucial point. There are at least two ways to approach variance (which are sort of dual to one another):
(1) I need to measure the spread of a distribution of values. What measure should I use? [Possible answer: variance.]
-or-
(2) If I define V[X] = E[(X-E[X])^2], then V[X] has many nice properties and seems to relate well to other parts of the theory and even other parts of mathematics. Obviously, it's something pretty important theoretically. What are its practical uses? [Possible answer: as a measure of spread.]
If the only thing you care about is measuring spread, then convenience may be the only reason for you to use the variance, if you use it at all. I don't think anybody can seriously claim that among all measures of spread, the variance is absolutely the best-quality measure in all situations. Yes, the variance is additive and allows you to formulate the central limit theorem, and properties like that are certainly very nice to have, both in theory and in practice, but they don't make it a better measure of spread. So from this perspective, "convenience" often is the right answer.
On the other hand, as a theorist, you would probably want to develop the theory along the most fruitful route, so you would be stupid to ignore the variance in any case. Its usefulness as a measure of spread then is less important, and its overall properties are rather more important. From this perspective, "convenience" isn't really the right answer, since it doesn't convey the intrinsic value it has by virtue of the excellent theory surrounding it.
The question seems to be asked from a statistics point of view.
In statistics, (sample) standard deviation is used as a measure of dispersion in data. It does have many nice properties, but as others have said, why we choose it is often due to convenience.
The convenience stems from the fact that we often want to minimize dispersion. With standard deviation, such minimization problems tend to have explicit solutions, while for other dispersion measures numeric methods has to be used to a much higher degree. Also, explicit solutions simplify statistical theory a lot.
It does have its drawbacks however, and is not the only dispersion measure used in practice. A big drawback is that it is sensitive to outliers in data. If to the data set {1,2,2,3,5}, the point 100 is added, the standard deviation changes a lot. It is not the best measure for highly skewed data, or data from distributions with heavy tails.
One alternative is the interquartile range, the distance between the 3rd and 1st quartiles, and mean absolute distance to the median is another.
See the wikipedia article on Robust Statistics for more information.
Here is a simple-minded explanation: The standard deviation as a "measure of dispersion" is the natural partner of the arithmetic mean as a "central statistic".
Suppose we are given $n+1$ measurements (say lengths) $x_0 \le x_1 \le \cdots \le x_{n},$ and wish to choose a single value $x^*$ to represent them. We need a metric for how good a particular $x^*$ is. Then we choose the value which minimizes the "agregate discrepancy".
If our metric is $\sum |x_i-x^*|,$ then it is best to take $x^*=x_{n/2}$ (the median) for even $n$ and any $x_{(n-1)/2} \le x^* \le x_{(n+1)/2}$ for odd $n$. It is perhaps unfortunate that only one or two of the $x_i$ actually matter.
Of course for $\sum(x_i-x^*)^2$ the unique minimum occurs for the familiar arithmetic mean $x^*=\frac{\sum x_i}{n+1}.$ We prefer to use the metric $\sqrt{\sum(x_i-x^*)^2}$ Since the "dispersion" is the same for measuring in inches as in feet (and the units are correct). There are also reasons to divide by $n+1$ or by $n,$ but none of this changes the minimizing value and the question was about the squaring.
For $\sum|x_i-x^*|^p$ ( or better $\left(\sum|x_i-x^*|^p\right)^{1/p}$ ) with varying $p$ we get the standard median as $p \rightarrow 1^+$ and $\frac{x_0+x_n}2$ as $p \rightarrow \infty.$
I suppose the mode would result from calling the discrepancy $0$ or $1$ according as $x_i = x^*$ or $x_i \neq x^*.$
Would $\sum \ln|x_i-x^*|$ (equivalently, $e^{\sum \ln|x_i-x^*|}$) give the geometric mean $\sqrt[n+1]{\prod{x_i}}?$
It might not be hard to find other metrics which yield the harmonic mean $$\frac1{\sum \frac1{x_i}},$$ and perhaps even the AGM.
Think of the mean/expectation $m$ as the number for which some definition of $\text{variance}(\{x_i-m\})$ is minimized. The $m$ which achieves the minimum of a quadratic variance is the unique solution of a linear equation. Non quadratic variances result in hard to compute, and sometimes non-unique, means/expectations. Therefore one could say the answer is that 2=1+1.
With standard deviation as defined, you get cool results like Chebyshev's Theorem: for any distribution and k>1, at most 1/k^2 of the data fall outside of k standard deviations from the mean. So, for example, for any distribution at most a quarter of the data lies farther then two standard deviations from the mean, and at most 12% lie further than three standard deviations.
This and other theoretical advantages come from the long answer that Mark4483 gave. These things are important for developing inference models.
Population variance, E([X-E(X)]^2), can quite easily be estimated in an unbiased manner by the sample variance, (n-1)^{-1}\sum (X_i-\bar{X})^2, where the sum is from i=1 to i=n. "Unbiased" meaning that if the X_i are i.i.d. copies of X, then the expectation of the sample variance (by a straightforward direct computation) is the population variance.
As far as I know, you can't do anything at all for absolute values. For even powers larger than 2, a more complicated unbiased estimator could be constructed. Two is the simplest case, and plenty of interest happens - the central limit theorem, for starters. I'm not sure to what extent the L^2 theory generalizes to L^2k theory, but to my knowledge no compelling reason to consider it exists. Perhaps with higher k, the large deviations are weighted even more heavily, and particular applications may benefit from their study. However, the usual variance is a good starting point. There are conditions under which you can recover a random variable X if you know all of its moments, E(X^p) for p = 1, 2, 3, .... From this perspective, one might consider the variance, E(X^2)-E(X)^2, a function of the first two moments, for the same reason that one may look at a second-degree Taylor approximation of sin - it's the simplest, and you can look at other terms if you need to know more.
If you are dealing with negative numbers, you need to square them (or use absolute value) to capture the appropriate distance from the mean.
Using a square allows one to have a continuously differentiable function (absolute value is not differentiable at 0). So squaring is slightly more convenient than absolute value.
The final square root gets you back to your original units.
I think that is quite obvious.
Firstly, the sampling standard deviation measures the dispersion of the data in the samples.
You square the difference is to reflect how much the actually data is away from the sample mean. If you do not square them, some differences may cancel out each other e.g. -1, +1, in which case the dispersion cannot be reflected.
You square root it is for matching the dimension
