Short answer:  You could argue that the most natural thing to do when defining a "standard deviation-type" quantity is to use an absolute value:  $E(|X|)$,   but it's really annoying to deal with absolute values under the expectation, so we use the next best thing:  $\sqrt{E( X^2 )}$.  You still get something positive and it's easier to deal with the square inside. We take a square root at the end to get something with the same "units" as $X$. 


Long answer:  It's often helpful to think of random variables as living in the function space $L^2(\Omega)$,  and in this setting, this computation gives the $L^2$ norm of the centered random variable $X - EX$.  Also, with this perspective,  the covariance defines an inner product.