Short answer: You could argue that the most natural thing to do when defining a "standard deviation-type" quantity is to use an absolute value: $E(|X|)$, but it's really annoying to deal with absolute values under the expectation, so we use the next best thing: $\sqrt{E( X^2 )}$. You still get something positive and it's easier to deal with the square inside. We take a square root at the end to get something with the same "units" as $X$.
Long answer: It's often helpful to think of random variables as living in the function space $L^2(\Omega)$, and in this setting, this computation gives the $L^2$ norm of the centered random variable $X - EX$. Also, with this perspective, the covariance defines an inner product.
