46
$\begingroup$

When we want to find the standard deviation of $\{1,2,2,3,5\}$ we do

$$\sigma = \sqrt{ {1 \over 5-1} \left( (1-2.6)^2 + (2-2.6)^2 + (2-2.6)^2 + (3-2.6)^2 + (5 - 2.6)^2 \right) } \approx 1.52$$.

Why do we need to square and then square-root the numbers?

$\endgroup$
9
  • 3
    $\begingroup$ +1, excellent question, though I think the title can be improved. I think you selected the wrong answer though — there are many better reasons for the deviation formula than convenience. $\endgroup$ Oct 22, 2009 at 18:07
  • 9
    $\begingroup$ I hope that (over four years later) someone will return to this classical question. There seems to be something special about the number $p=2$ in $L_p$ norms. What is it? Flatness? Something exotic? (whatever makes Gleason's theorem work?) It seems to me that whatever makes $2$ special is independent of the CLT. It would be disappointing if the Gaussian assumption is the final reason for $p=2$. $\endgroup$ Jan 14, 2014 at 16:36
  • 2
    $\begingroup$ In the formula for $\sigma$ above, the term $(2-2.6)^2$ should of course appear twice. $\endgroup$ Mar 22, 2015 at 15:16
  • $\begingroup$ I've taken the liberty of inserting the missing term. ${}\qquad{}$ $\endgroup$ Mar 22, 2015 at 16:24
  • 3
    $\begingroup$ @Avicenna : I think your comment is silly. Obviously the sum of all deviations from the mean is zero. The obvious alternative to squaring the deviations is not just leaving them alone; it's taking their absolute values, since those are the distances from the observations to the mean. $\endgroup$ Mar 23, 2015 at 18:34

13 Answers 13

-4
$\begingroup$

One answer is mathematical convenience. The theory is much simpler using powers of two rather than other powers.

There are justifications. Squaring makes small numbers smaller and makes big numbers bigger. You could argue that a useful measure of dispersion should be forgiving of small errors but weigh larger errors more heavily, so it makes sense to square the deviations from the mean.

$\endgroup$
1
  • 2
    $\begingroup$ Sorry for -1, but you miss the key point, see my answer. $\endgroup$ Oct 22, 2009 at 18:04
54
$\begingroup$

Intro by Reid Barton

I think the answer should involve the additivity of variance for independent variables and the central limit theorem. Maybe someone can flesh this out.

Answer

Indeed, the variance has the additive property: if $r_1$ and $r_2$ are random variables with means $\mu_1, \mu_2$ and variances $d_1, d_2$, and these two variables are independent, then the new random variable $r = r_1+r_2$ has the mean $\mu_1+\mu_2$ and variance $d_1+d_2$.

Moreover, suppose we sum a large number $N$ of independent copies of our random variable $r$ with mean $\mu$ and variance $d$. Under mild assumptions, the central limit says the distribution will approach a normal distribution, which by the above has mean $N\mu$ and variance $Nd$. Observe that a normal distribution is completely determined by its mean and variance. We conclude that the only parameters of a distribution that we can observe from the sum of many independent copies of the distribution are the mean and variance.

Now that we established how good it is to square numbers, to get variance, the standard deviation has a very easy explanation: it's the only way to get back from variance to something with the dimension of our original set. That is, suppose you numbers are some lengths written in meters. Since the variance is meters squared, you have to take the square root to get something that can be compared with the original set.

Now, honestly, this not the only way, since you could also, e.g., multiply it by 2. That's why it's called standard deviation — to show that among different numerical constants we've chosen a specific one.

$\endgroup$
11
  • 3
    $\begingroup$ Excellent use of community wiki! $\endgroup$ Oct 19, 2009 at 7:00
  • $\begingroup$ Ilya seems to have confused variance and standard deviation. The variance of the sum of independent random variables is the sum of the variances: Var(X+Y)=Var(X)+Var(Y) so StdDev(X+Y)=sqrt(Var(X)+Var(Y)), which in general is not equal to StdDev(X)+StdDev(Y) $\endgroup$ Oct 22, 2009 at 18:22
  • $\begingroup$ Fixed, and included the relation to the central limit theorem. $\endgroup$ Oct 22, 2009 at 18:41
  • 18
    $\begingroup$ I think you guys are dismissing the L^2 interpretation of this question. Variance is important, but it is just a special case of covariance, which is where the real action is going on. All the nice properties of variance follow from the fact that covariance is more or less an inner product, and independent random variables are orthogonal with respect to that inner product. $\endgroup$ Oct 24, 2009 at 19:27
  • 3
    $\begingroup$ I think the additivity of variance for independent r.v.s is not the sole raison d'etre here. Consider the 3rd centred moment $T(X)=E( (X-EX)^3 )$. Then if X and Y are independent, with respective means m_x and m_y say, then T(X+Y) = E((X-m_x+Y-m_y)^3)) = E(X-m_x)^3 + E(Y-m_y)^3 = T(X) + T(Y). In the same vein, see the wiki entry for "cumulant". The argument in terms of CLT and parameters of the limit distribution is good though. $\endgroup$
    – Yemon Choi
    Oct 28, 2009 at 4:03
22
$\begingroup$

Short answer: You could argue that the most natural thing to do when defining a "standard deviation-type" quantity is to use an absolute value: $E(|X|)$, but it's really annoying to deal with absolute values under the expectation, so we use the next best thing: $\sqrt{E( X^2 )}$. You still get something positive and it's easier to deal with the square inside. We take a square root at the end to get something with the same "units" as $X$.

Long answer: It's often helpful to think of random variables as living in the function space $L^2(\Omega)$, and in this setting, this computation gives the $L^2$ norm of the centered random variable $X - EX$. Also, with this perspective, the covariance defines an inner product.

$\endgroup$
7
  • 1
    $\begingroup$ Sorry, I think I was too rush in downvoting. The L^2 observation makes a lot of sense and is quite deep. $\endgroup$ Oct 24, 2009 at 20:31
  • 33
    $\begingroup$ What definitions are you using for "short" and "long"? $\endgroup$
    – Cap Khoury
    Jul 30, 2010 at 1:26
  • 5
    $\begingroup$ But why is $L^2$ cooler than $L^4$ or some other absolute value/function space? $\endgroup$ Jan 14, 2014 at 16:17
  • $\begingroup$ @isomorphismes It has more random variables with finite norm. And it implies that two random variables with finite variance have also a finite covariance. $\endgroup$ Jan 14, 2014 at 17:55
  • 8
    $\begingroup$ The longer answer is shorter than the short answer. :) $\endgroup$
    – Rasmus
    Mar 22, 2015 at 11:02
19
$\begingroup$

If you apply Bessel's correction — dividing by $5-1$ rather than by $5$ when you have $5$ numbers — then some of the otherwise right things stated in some of the answers are wrong. Bessel's correction is intended to be used only when the variance one is computing is based on a sample to be used to estimate the variance of the whole population.

I won't be surprised if nobody used the variance and standard deviation before Abraham de Moivre did so in the 18th century. De Moivre considered this question: If you toss a fair coin $1800$ times, what is the probability that the number of heads is in some specified range? You have a binomial distribution, and computing its exact values was not feasible. De Moivre approximated the distribution of the number of heads with a normal distribution with the same mean and the same standard deviation. In so doing, he was the first to introduce the normal distribution, and the first to prove a special case of the central limit theorem. The normal distribution with mean $0$ and variance $1$ is $$ \varphi(x)\,dx=\frac 1 {\sqrt{2\pi}} e^{-x^2/2}\,dx $$ and with mean $\mu$ and variance $\sigma^2$ it is $$ \varphi\left(\frac{x-\mu}\sigma\right)\, \frac{dx}\sigma. $$ It's easy to find the mean and standard deviation for the number of heads when one fair coin is tossed: they're both $1/2$. How do you do it for the sum of $1800$ independent copies of that random variable? De Moivre found that the mean-square deviation is additive: for independent random variables $X_1,\dotsc,X_{1800}$ one has $\operatorname{var}(X_1+\dotsb+X_{1800})=\operatorname{var}(X_1)+\dotsb+\operatorname{var}(X_{1800})$. You cannot do that with mean absolute deviation. If I'm recalling some details correctly, he published these findings in a paper in Latin while he lived in France, and at that time he gave the normal distribution as $$ C e^{-x^2/2}\,dx $$ where he could find $C$ only numerically. Later he went to England to escape the persecution of Protestants and met James Stirling, who showed that $C=1/\sqrt{2\pi}$. De Moivre wrote a book in English called The Doctrine of Chances, which I think was 18th-century English for the theory of probability. Some have speculated that the Reverend Thomas Bayes may have studied under him, but I don't know that that's gone beyond speculation.

(If you want to know the probability that the number of heads is $\ge894$, note that that's the same as “${>893}$”, and find the probability that the normally distributed random variable with the same mean and variance is $>893.5$. That is a "continuity correction" and works surprisingly well even for fairly small samples.)

On to Bessel's correction: When does one use $$ \frac{(x_1-\bar x)^2+\cdots+(x_n-\bar x)^2}{n-1}, $$ where $\bar x=(x_1+\cdots+x_n)/n$, with $n-1$ rather than $n$ in the denominator? As you can see from simple examples, that will not serve de Moivre's purpose described above: it's not additive.

If $X_1,\dotsc,X_n$ are an independent sample from a population with mean $\mu$ and variance $\sigma^2$, then the expected value of $$ \frac{(X_1-\mu)^2+\dotsb+(X_n-\mu)^2} n \tag 1\label{1} $$ is $\sigma^2$. But if one has only the sample and not the whole population, one doesn't know $\mu$ and one can use the sample mean $\bar X$ as an estimate of $\mu$. But the expected value of $$ \frac{(X_1-\bar X)^2+\dotsb+(X_n-\bar X)^2} n $$ is smaller than the expected value of \eqref{1}. Specifically, a bit of algebra shows that $$ \sum_{i=1}^n (X_i-\mu)^2 = \left( \sum_{i=1}^n (X_i-\bar X)^2 \right) + n(\bar X-\mu)^2, \tag 2\label{2} $$ and since the expectation of the last term is $\sigma^2$, that of the first term on the right in \eqref{2} must be $(n-1)\sigma^2$. Thus Bessel's correction gives an unbiased estimate of the population variance $\sigma^2$. (But its square root does not give an unbiased estimate of the population standard deviation. And unbiasedness is at best somewhat overrated anyway, and in some cases is a very very bad thing (I had a paper in the American Mathematical Monthly a few years ago demonstrating how bad it can sometimes be).

$\endgroup$
12
$\begingroup$

One answer I've heard is that you want the notion of standard deviation to 1) apply to points in Euclidean space, and 2) to be invariant under rotation. You don't get the second property unless you square the distances.

$\endgroup$
3
  • 7
    $\begingroup$ What does "rotation" mean in statistics? It's clear in physics why we should prize rotation-invariance, but if our three measurements of a person are the length of their surname + their height + their wealth, then I would only compute SD 1-dimensionally. $\endgroup$ Jan 14, 2014 at 16:22
  • 3
    $\begingroup$ I guess PCA involves rotation... $\endgroup$ Jan 16, 2014 at 4:32
  • $\begingroup$ @isomorphismes : Far more than just PCA involves rotation in statistics. $\endgroup$ Apr 1, 2023 at 22:18
12
$\begingroup$

The answers here that stress that "the answers here that stress convenience are missing the crucial point" are missing a crucial point. There are at least two ways to approach variance (which are sort of dual to one another):

(1) I need to measure the spread of a distribution of values. What measure should I use? [Possible answer: variance.]

-or-

(2) If I define V[X] = E[(X-E[X])^2], then V[X] has many nice properties and seems to relate well to other parts of the theory and even other parts of mathematics. Obviously, it's something pretty important theoretically. What are its practical uses? [Possible answer: as a measure of spread.]

If the only thing you care about is measuring spread, then convenience may be the only reason for you to use the variance, if you use it at all. I don't think anybody can seriously claim that among all measures of spread, the variance is absolutely the best-quality measure in all situations. Yes, the variance is additive and allows you to formulate the central limit theorem, and properties like that are certainly very nice to have, both in theory and in practice, but they don't make it a better measure of spread. So from this perspective, "convenience" often is the right answer.

On the other hand, as a theorist, you would probably want to develop the theory along the most fruitful route, so you would be stupid to ignore the variance in any case. Its usefulness as a measure of spread then is less important, and its overall properties are rather more important. From this perspective, "convenience" isn't really the right answer, since it doesn't convey the intrinsic value it has by virtue of the excellent theory surrounding it.

$\endgroup$
6
  • 1
    $\begingroup$ Interesting (and +1) $\endgroup$ Oct 24, 2009 at 20:27
  • 3
    $\begingroup$ But no, I think you're stepping on a bit confusing path. Yes, if you just define the variance, it is convenient. But why is it convenient? You should say something about the reason, otherwise you're just changing a mystery as for why to use variance into a mystery as for why variance is convenient. $\endgroup$ Oct 24, 2009 at 20:30
  • $\begingroup$ Though I agree may be I should have been more respectful to L^2 argument. $\endgroup$ Oct 24, 2009 at 20:31
  • 1
    $\begingroup$ @Martin Hairer: I think the duplicate words in the first sentence were intentional. Darsh Ranjan seems to be saying that people who say "People are missing Point A" are in fact themselves missing Point B. $\endgroup$ Jul 9, 2015 at 9:16
  • $\begingroup$ @potentially dense: Oops, you are right, I was being too quick. But there is surely a less confusing way to say this... $\endgroup$ Jul 9, 2015 at 9:56
6
$\begingroup$

The question seems to be asked from a statistics point of view.

In statistics, (sample) standard deviation is used as a measure of dispersion in data. It does have many nice properties, but as others have said, why we choose it is often due to convenience.

The convenience stems from the fact that we often want to minimize dispersion. With standard deviation, such minimization problems tend to have explicit solutions, while for other dispersion measures numeric methods has to be used to a much higher degree. Also, explicit solutions simplify statistical theory a lot.

It does have its drawbacks however, and is not the only dispersion measure used in practice. A big drawback is that it is sensitive to outliers in data. If to the data set {1,2,2,3,5}, the point 100 is added, the standard deviation changes a lot. It is not the best measure for highly skewed data, or data from distributions with heavy tails.

One alternative is the interquartile range, the distance between the 3rd and 1st quartiles, and mean absolute distance to the median is another.

See the wikipedia article on Robust Statistics for more information.

$\endgroup$
4
$\begingroup$

Here is a simple-minded explanation: The standard deviation as a "measure of dispersion" is the natural partner of the arithmetic mean as a "central statistic".

Suppose we are given $n+1$ measurements (say lengths) $x_0 \le x_1 \le \cdots \le x_{n},$ and wish to choose a single value $x^*$ to represent them. We need a metric for how good a particular $x^*$ is. Then we choose the value which minimizes the "agregate discrepancy".

  • If our metric is $\sum |x_i-x^*|,$ then it is best to take $x^*=x_{n/2}$ (the median) for even $n$ and any $x_{(n-1)/2} \le x^* \le x_{(n+1)/2}$ for odd $n$. It is perhaps unfortunate that only one or two of the $x_i$ actually matter.

  • Of course for $\sum(x_i-x^*)^2$ the unique minimum occurs for the familiar arithmetic mean $x^*=\frac{\sum x_i}{n+1}.$ We prefer to use the metric $\sqrt{\sum(x_i-x^*)^2}$ Since the "dispersion" is the same for measuring in inches as in feet (and the units are correct). There are also reasons to divide by $n+1$ or by $n,$ but none of this changes the minimizing value and the question was about the squaring.

  • For $\sum|x_i-x^*|^p$ with varying $p$ we get the standard median as $p \rightarrow 1^+$ and $\frac{x_0+x_n}2$ as $p \rightarrow \infty.$

  • I suppose the mode would result from calling the discrepancy $0$ or $1$ according as $x_i = x^*$ or $x_i \neq x^*.$

  • Would $\sum \ln|x_i-x^*|$ (equivalently, $e^{\sum \ln|x_i-x^*|}$) give the geometric mean $\sqrt[n+1]{\prod{x_i}}?$

  • It might not be hard to find other metrics which yield the harmonic mean $$\frac1{\sum \frac1{x_i}},$$ and perhaps even the AGM.

$\endgroup$
1
  • 1
    $\begingroup$ You get the geometric mean from $\sum_i(\log(x^*/x_i))^2$ and the harmonic mean from $\sum_i(1/x^* - 1/x_i)^2$. A similar result holds for any generalized mean of the form $f(\sum_i f^{-1}(x_i)/(n+1))$. AGM is interesting, though...I don't know about that one. $\endgroup$ Jul 15, 2015 at 21:43
1
$\begingroup$

Think of the mean/expectation $m$ as the number for which some definition of $\text{variance}(\{x_i-m\})$ is minimized. The $m$ which achieves the minimum of a quadratic variance is the unique solution of a linear equation. Non quadratic variances result in hard to compute, and sometimes non-unique, means/expectations. Therefore one could say the answer is that 2=1+1.

$\endgroup$
1
$\begingroup$

With standard deviation as defined, you get cool results like Chebyshev's Theorem: for any distribution and k>1, at most 1/k^2 of the data fall outside of k standard deviations from the mean. So, for example, for any distribution at most a quarter of the data lies farther then two standard deviations from the mean, and at most 12% lie further than three standard deviations.

This and other theoretical advantages come from the long answer that Mark4483 gave. These things are important for developing inference models.

$\endgroup$
1
  • 7
    $\begingroup$ If we define the standard deviation with absolute value instead of squares, we would still have Chebyshev's theorem except with 1/k in place of 1/k^2 $\endgroup$
    – Yoo
    Nov 5, 2009 at 7:57
0
$\begingroup$

Population variance, E([X-E(X)]^2), can quite easily be estimated in an unbiased manner by the sample variance, (n-1)^{-1}\sum (X_i-\bar{X})^2, where the sum is from i=1 to i=n. "Unbiased" meaning that if the X_i are i.i.d. copies of X, then the expectation of the sample variance (by a straightforward direct computation) is the population variance.

As far as I know, you can't do anything at all for absolute values. For even powers larger than 2, a more complicated unbiased estimator could be constructed. Two is the simplest case, and plenty of interest happens - the central limit theorem, for starters. I'm not sure to what extent the L^2 theory generalizes to L^2k theory, but to my knowledge no compelling reason to consider it exists. Perhaps with higher k, the large deviations are weighted even more heavily, and particular applications may benefit from their study. However, the usual variance is a good starting point. There are conditions under which you can recover a random variable X if you know all of its moments, E(X^p) for p = 1, 2, 3, .... From this perspective, one might consider the variance, E(X^2)-E(X)^2, a function of the first two moments, for the same reason that one may look at a second-degree Taylor approximation of sin - it's the simplest, and you can look at other terms if you need to know more.

$\endgroup$
0
$\begingroup$

If you are dealing with negative numbers, you need to square them (or use absolute value) to capture the appropriate distance from the mean.

Using a square allows one to have a continuously differentiable function (absolute value is not differentiable at 0). So squaring is slightly more convenient than absolute value.

The final square root gets you back to your original units.

$\endgroup$
2
  • $\begingroup$ The square-root-of-the-square function is also not differentiable at $0$; in fact, in 1 dimension, all the various $L^p$ norms coincide. $\endgroup$
    – LSpice
    Mar 6, 2023 at 15:31
  • $\begingroup$ @LSpice you are correct. After taking the final square, the function is not differentiable at 0. $\endgroup$
    – devdrc
    Mar 6, 2023 at 15:40
-3
$\begingroup$

I think that is quite obvious.

Firstly, the sampling standard deviation measures the dispersion of the data in the samples.

You square the difference is to reflect how much the actually data is away from the sample mean. If you do not square them, some differences may cancel out each other e.g. -1, +1, in which case the dispersion cannot be reflected.

You square root it is for matching the dimension

$\endgroup$
1
  • 5
    $\begingroup$ There are more direct ways to deal with canceling, such as averaging the magnitude of the difference. $\endgroup$
    – user13113
    Sep 10, 2011 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.