Short answer: You could argue that the most natural thing to do when defining a "standard deviation-type" quantity is to use an absolute value: E( |X| ), but its really annoying to deal with absolute values under the expectation, so we use the next best thing: E( X^2 )^(1/2). You still get something positive and its easier to deal with the square inside. We take a square root at the end to get something with the same "units" as X.
Long answer: It's often helpful to think of random variables as living in the function space L^2(Omega), and in this setting, this computation gives the L^2 norm of the centered random variable X - EX. Also, with this perspective, the covariance defines is a inner product.