# expectation and variance of the norm of a random matrix

Suppose $$X \in \mathbb{R}^{n \times d}$$ is a random matrix where $$n > d$$. Given a matrix $$A \in \mathbb{R}^{n \times n}$$ such that $$AX$$ is a zero matrix in expectation, i.e., $$\mathbb{E}_{X}[AX] = 0$$. Let $$\sigma^2$$ be the variance of the norm of $$AX$$, i.e., $$\sigma^2:=\mathbb{V}[\lVert AX \rVert^2_F]$$.

Now I would like to study the property of the random matrix $$B=X(X^\top X)^{-1}X^\top AX$$. Do we also have $$\mathbb{E}[B] = 0$$ and can we bound $$\mathbb{V}[\lVert B \rVert^2_F]$$ by $$\sigma^2$$?

$$\newcommand\si\sigma\newcommand\bm[1]{\begin{bmatrix}#1\end{bmatrix}}$$No and no: In general, (i) $$EB\ne0$$ and (ii) we cannot bound $$Var\,\|B\|_F^2$$ by $$\si^2$$.

E.g., let $$n=3$$, $$d=1$$, $$y_1:=\bm{1\\ -1\\ -1},\quad y_2:=\bm{-1\\ 1\\ -1},\quad y_3:=\bm{-1\\ -1\\ 1},\quad y_4:=\bm{1\\ 1\\ 1},$$ $$A:=\bm{1&-1&0\\0&1&0\\0&0&1}.$$ Let $$Y$$ be a random matrix such that $$P(Y=y_j)=1/4$$ for $$j=1,2,3,4$$. Let $$X:=A^{-1}Y.$$

Then $$EAX=EY=0$$ and $$\|AX\|_F^2=\|Y\|_F^2=3$$ almost surely (a.s.), so that $$\si^2=Var\,\|AX\|_F^2=0.$$

However, $$EB=\bm{0\\ 0\\ -1/6}\ne0.$$ Also, the values of $$\|B\|_F^2$$ at $$Y=y_1$$ and at $$Y=y_4$$ are the non-equal numbers $$2$$ and $$8/3$$, respectively, so that $$Var\,\|B\|_F^2$$ is strictly greater than $$0=\si^2$$. (In fact, $$Var\,\|B\|_F^2=1/9$$.)

One might feel some affinity for the OP's conjectures. Indeed,

(i) we have $$B=P_X AX$$, where $$P_X$$ is the matrix of the orthoprojector onto the column space of the matrix $$X$$. So, if $$P_X$$ did not depend on $$X$$, the conjectured conclusion $$EB=0$$ would follow from $$EAX=0$$ and the linearity of any orthoprojector. Also,

(ii) since $$P_X$$ is the matrix of an orthoprojector, the equality $$B=P_X AX$$ does imply $$\|B\|_F^2\le\|AX\|_F^2$$. However, as shown above, this will not in general imply that $$Var\,\|B\|_F^2\le Var\,\|AX\|_F^2$$.

• Thanks for your solution and in-depth discussion. For the OP's conjectures part, I am not sure if I understand what you mean by $P_X$ did not depend on $X$. Why $P_X$ will not depend on $X$? It is an assumption or sometimes with certain distribution on $X$, $P_X$ may become independent on $X$. Further, I would like to ask, in my research I could assume that entries of $X$ have i.i.d Gaussian distributions. Will this extra condition change the situation? Commented Mar 20, 2023 at 1:33
• @HaoHe : "if $P_X$ did not depend on $X$" is in the subjunctive mood. Of course, usually $P_X$ will in fact depend on $X$. However, if $X$ is zero-mean Gaussian (or any symmetric random matrix), then, noting that $P_{-X}A(-X)=-P_X AX$, we see that $B$ is symmetric and hence $EB=0$ (if $EB$ exists, which will of course exist if $X$ is Gaussian). Commented Mar 20, 2023 at 2:15
• Thank you for the follow-up answer. I can now see that $EB=0$. How about $Var \| B \|^2_F$? Will we be able to bound it? or know something about it? Commented Mar 20, 2023 at 3:02
No, you cannot conclude that $$\mathbb{E}[B]=0$$. Here is a counterexample for $$n=2,d=1$$: $$A={{1\,1}\choose{0\,0}}$$, $$X={a\choose -b}$$, with $$\mathbb{E}[a]=\mathbb{E}[b]$$. Then $$AX={a-b\choose 0}$$ satisfies $$\mathbb{E}[AX]=0$$, however the matrix $$B=X(X^\top X)^{-1}X^\top AX=\frac{a^2-ab}{a^2+b^2}{a\choose -b}$$ does not necessarily have a vanishing expectation value (for example, $$\mathbb{E}[B]\neq 0$$ if $$\mathbb{E}[a^2b]\neq\mathbb{E}[ab^2]$$).