2
$\begingroup$

Suppose $X \in \mathbb{R}^{n \times d}$ is a random matrix where $n > d$. Given a matrix $A \in \mathbb{R}^{n \times n}$ such that $AX$ is a zero matrix in expectation, i.e., $\mathbb{E}_{X}[AX] = 0$. Let $\sigma^2$ be the variance of the norm of $AX$, i.e., $\sigma^2:=\mathbb{V}[\lVert AX \rVert^2_F]$.

Now I would like to study the property of the random matrix $B=X(X^\top X)^{-1}X^\top AX$. Do we also have $\mathbb{E}[B] = 0$ and can we bound $\mathbb{V}[\lVert B \rVert^2_F]$ by $\sigma^2$?

$\endgroup$

2 Answers 2

1
$\begingroup$

$\newcommand\si\sigma\newcommand\bm[1]{\begin{bmatrix}#1\end{bmatrix}}$No and no: In general, (i) $EB\ne0$ and (ii) we cannot bound $Var\,\|B\|_F^2$ by $\si^2$.

E.g., let $n=3$, $d=1$, $$y_1:=\bm{1\\ -1\\ -1},\quad y_2:=\bm{-1\\ 1\\ -1},\quad y_3:=\bm{-1\\ -1\\ 1},\quad y_4:=\bm{1\\ 1\\ 1}, $$ $$A:=\bm{1&-1&0\\0&1&0\\0&0&1}.$$ Let $Y$ be a random matrix such that $P(Y=y_j)=1/4$ for $j=1,2,3,4$. Let $$X:=A^{-1}Y.$$

Then $EAX=EY=0$ and $\|AX\|_F^2=\|Y\|_F^2=3$ almost surely (a.s.), so that $$\si^2=Var\,\|AX\|_F^2=0.$$

However, $$EB=\bm{0\\ 0\\ -1/6}\ne0.$$ Also, the values of $\|B\|_F^2$ at $Y=y_1$ and at $Y=y_4$ are the non-equal numbers $2$ and $8/3$, respectively, so that $Var\,\|B\|_F^2$ is strictly greater than $0=\si^2$. (In fact, $Var\,\|B\|_F^2=1/9$.)


One might feel some affinity for the OP's conjectures. Indeed,

(i) we have $B=P_X AX$, where $P_X$ is the matrix of the orthoprojector onto the column space of the matrix $X$. So, if $P_X$ did not depend on $X$, the conjectured conclusion $EB=0$ would follow from $EAX=0$ and the linearity of any orthoprojector. Also,

(ii) since $P_X$ is the matrix of an orthoprojector, the equality $B=P_X AX$ does imply $\|B\|_F^2\le\|AX\|_F^2$. However, as shown above, this will not in general imply that $Var\,\|B\|_F^2\le Var\,\|AX\|_F^2$.

$\endgroup$
4
  • $\begingroup$ Thanks for your solution and in-depth discussion. For the OP's conjectures part, I am not sure if I understand what you mean by $P_X$ did not depend on $X$. Why $P_X$ will not depend on $X$? It is an assumption or sometimes with certain distribution on $X$, $P_X$ may become independent on $X$. Further, I would like to ask, in my research I could assume that entries of $X$ have i.i.d Gaussian distributions. Will this extra condition change the situation? $\endgroup$
    – Hao He
    Commented Mar 20, 2023 at 1:33
  • 1
    $\begingroup$ @HaoHe : "if $P_X$ did not depend on $X$" is in the subjunctive mood. Of course, usually $P_X$ will in fact depend on $X$. However, if $X$ is zero-mean Gaussian (or any symmetric random matrix), then, noting that $P_{-X}A(-X)=-P_X AX$, we see that $B$ is symmetric and hence $EB=0$ (if $EB$ exists, which will of course exist if $X$ is Gaussian). $\endgroup$ Commented Mar 20, 2023 at 2:15
  • $\begingroup$ Thank you for the follow-up answer. I can now see that $EB=0$. How about $Var \| B \|^2_F$? Will we be able to bound it? or know something about it? $\endgroup$
    – Hao He
    Commented Mar 20, 2023 at 3:02
  • $\begingroup$ @HaoHe : Please consider the following. Asking multiple questions in one post is discouraged on MathOverflow. You asked two questions in your post on this page. I answered both. In addition, I provided a discussion of those answers, which you apparently found useful. Moreover, I answered the extra question in your comment. So, to keep things in good order, can we now wrap up this matter? You may want to post your further questions separately, which will also give them greater exposure. $\endgroup$ Commented Mar 20, 2023 at 12:41
1
$\begingroup$

No, you cannot conclude that $\mathbb{E}[B]=0$. Here is a counterexample for $n=2,d=1$: $A={{1\,1}\choose{0\,0}}$, $X={a\choose -b}$, with $\mathbb{E}[a]=\mathbb{E}[b]$. Then $AX={a-b\choose 0}$ satisfies $\mathbb{E}[AX]=0$, however the matrix $$B=X(X^\top X)^{-1}X^\top AX=\frac{a^2-ab}{a^2+b^2}{a\choose -b}$$ does not necessarily have a vanishing expectation value (for example, $\mathbb{E}[B]\neq 0$ if $\mathbb{E}[a^2b]\neq\mathbb{E}[ab^2]$).

$\endgroup$
1
  • $\begingroup$ Thanks for your counter-example, Carlo. $\endgroup$
    – Hao He
    Commented Mar 20, 2023 at 1:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.