What concept does covariance formalise?

Question

So for me the definition the independence of two random variables $X,Y$ is intuitivly very clear.

But what I have never seen motivated is why the heck one would be interested in the covariance $$\operatorname{Cov}(X,Y):=\mathbb{E}\left((X-\mathbb{E}(X))(Y-\mathbb{E}(Y))\right).$$ Since independent variables are also uncorrelated, covariance seems to be some weak form of independence.

Of course there are various nice results for uncorrelated random variables, for example $\mathbb{E}(XY)=\mathbb{E}(X)\mathbb{E}(Y)$ or various forms of convergence results like the law of large numbers or the central limit theorem. But what is the essence of covariance, what intuitive notion does it capture? Every example of uncorrelated yet dependent random variables seem very contrived to me. What would a "real-life-example" of two such variables look like?

Answer 1

$\DeclareMathOperator\Cov{Cov}\DeclareMathOperator\Var{Var}$I. "[W]hy the heck one would be interested in the covariance"? At least for two reasons:

1. The correlation $$\rho_{X,Y}\mathrel{:=}\frac{\Cov(X,Y)}{\sqrt{\Var X}\sqrt{\Var Y}}$$ is the normalized covariance (assuming $\Var X\ne0$ and $\Var Y\ne0$), with the properties that (i) $-1\le\rho_{X,Y}\le1$, (ii) $\rho_{X,Y}=1$ iff $Y=aX+b$ almost surely (a.s.) for some real $a>0$ and some real $b$, and (iii) $\rho_{X,Y}=-1$ iff $Y=aX+b$ a.s. for some real $a<0$ and some real $b$. So, $\rho_{X,Y}$ characterizes the strength of a linear relationship between $X$ and $Y$.

2. In contrast with the correlation, the covariance has the important bilinearity property: $$\Cov\Big(\sum_{i=1}^m a_iX_i,\sum_{j=1}^n b_jY_j\Big) =\sum_{i=1}^m \sum_{j=1}^n a_ib_j\Cov(X_i,Y_j)$$ for real $a_i,b_j$ (assuming finite second moments of the $X_i$'s and $Y_j$'s).

II. "Every example of uncorrelated yet dependent random variables seem very contrived to me." As was noted, the correlation characterizes the strength of a linear relationship between $X$ and $Y$. Here is an example of random variable (r.v.) $Y$ completely depending on, but uncorrelated with, a r.v. $X$: Let $X$ be any r.v. with $0<E\lvert X\rvert^3<\infty$ whose distribution is symmetric about $0$, and let $Y\mathrel{:=}X^2$. Then $\rho_{X,Y}=0$ and thus "there is no linear relationship" between $X$ and $Y$, but $Y$ is completely determined by $X$.

Answer 2

To add a few additional comments: Full statistical independence of two variables, say X and Y, is actually equivalent to the absence of correlation between f(X) and g(Y) for all functions f and g. And it is in fact enough to restrict the f and g to a `separating class' of functions, for example the sinusoids (for which we would not even need X and Y to have any moments).