2
$\begingroup$

Suppose $X \in \mathbb{R}^{n \times d}$ is a random matrix where $n > d$. Given a matrix $A \in \mathbb{R}^{n \times n}$ such that $AX$ is a zero matrix in expectation, i.e., $\mathbb{E}_{X}[AX] = 0$. Let $\sigma^2$ be the variance of the norm of $AX$, i.e., $\sigma^2:=\mathbb{V}[\lVert AX \rVert^2_F]$.

Now I would like to study the property of the random matrix $B=X(X^\top X)^{-1}X^\top AX$. Do we also have $\mathbb{E}[B] = 0$ and can we bound $\mathbb{V}[\lVert B \rVert^2_F]$ by $\sigma^2$?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

$\newcommand\si\sigma\newcommand\bm[1]{\begin{bmatrix}#1\end{bmatrix}}$No and no: In general, (i) $EB\ne0$ and (ii) we cannot bound $Var\,\|B\|_F^2$ by $\si^2$.

E.g., let $n=3$, $d=1$, $$y_1:=\bm{1\\ -1\\ -1},\quad y_2:=\bm{-1\\ 1\\ -1},\quad y_3:=\bm{-1\\ -1\\ 1},\quad y_4:=\bm{1\\ 1\\ 1}, $$ $$A:=\bm{1&-1&0\\0&1&0\\0&0&1}.$$ Let $Y$ be a random matrix such that $P(Y=y_j)=1/4$ for $j=1,2,3,4$. Let $$X:=A^{-1}Y.$$

Then $EAX=EY=0$ and $\|AX\|_F^2=\|Y\|_F^2=3$ almost surely (a.s.), so that $$\si^2=Var\,\|AX\|_F^2=0.$$

However, $$EB=\bm{0\\ 0\\ -1/6}\ne0.$$ Also, the values of $\|B\|_F^2$ at $Y=y_1$ and at $Y=y_4$ are the non-equal numbers $2$ and $8/3$, respectively, so that $Var\,\|B\|_F^2$ is strictly greater than $0=\si^2$. (In fact, $Var\,\|B\|_F^2=1/9$.)

One might feel some affinity for the OP's conjectures. Indeed,

(i) we have $B=P_X AX$, where $P_X$ is the matrix of the orthoprojector onto the column space of the matrix $X$. So, if $P_X$ did not depend on $X$, the conjectured conclusion $EB=0$ would follow from $EAX=0$ and the linearity of any orthoprojector. Also,

(ii) since $P_X$ is the matrix of an orthoprojector, the equality $B=P_X AX$ does imply $\|B\|_F^2\le\|AX\|_F^2$. However, as shown above, this will not in general imply that $Var\,\|B\|_F^2\le Var\,\|AX\|_F^2$.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for your solution and in-depth discussion. For the OP's conjectures part, I am not sure if I understand what you mean by $P_X$ did not depend on $X$. Why $P_X$ will not depend on $X$? It is an assumption or sometimes with certain distribution on $X$, $P_X$ may become independent on $X$. Further, I would like to ask, in my research I could assume that entries of $X$ have i.i.d Gaussian distributions. Will this extra condition change the situation? $\endgroup$
    – Hao He
    Commented Mar 20, 2023 at 1:33
  • 1
    $\begingroup$ @HaoHe : "if $P_X$ did not depend on $X$" is in the subjunctive mood. Of course, usually $P_X$ will in fact depend on $X$. However, if $X$ is zero-mean Gaussian (or any symmetric random matrix), then, noting that $P_{-X}A(-X)=-P_X AX$, we see that $B$ is symmetric and hence $EB=0$ (if $EB$ exists, which will of course exist if $X$ is Gaussian). $\endgroup$
    – Iosif Pinelis
    Commented Mar 20, 2023 at 2:15
  • $\begingroup$ Thank you for the follow-up answer. I can now see that $EB=0$. How about $Var \| B \|^2_F$? Will we be able to bound it? or know something about it? $\endgroup$
    – Hao He
    Commented Mar 20, 2023 at 3:02
  • $\begingroup$ @HaoHe : Please consider the following. Asking multiple questions in one post is discouraged on MathOverflow. You asked two questions in your post on this page. I answered both. In addition, I provided a discussion of those answers, which you apparently found useful. Moreover, I answered the extra question in your comment. So, to keep things in good order, can we now wrap up this matter? You may want to post your further questions separately, which will also give them greater exposure. $\endgroup$
    – Iosif Pinelis
    Commented Mar 20, 2023 at 12:41
1
$\begingroup$

No, you cannot conclude that $\mathbb{E}[B]=0$. Here is a counterexample for $n=2,d=1$: $A={{1\,1}\choose{0\,0}}$, $X={a\choose -b}$, with $\mathbb{E}[a]=\mathbb{E}[b]$. Then $AX={a-b\choose 0}$ satisfies $\mathbb{E}[AX]=0$, however the matrix $$B=X(X^\top X)^{-1}X^\top AX=\frac{a^2-ab}{a^2+b^2}{a\choose -b}$$ does not necessarily have a vanishing expectation value (for example, $\mathbb{E}[B]\neq 0$ if $\mathbb{E}[a^2b]\neq\mathbb{E}[ab^2]$).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your counter-example, Carlo. $\endgroup$
    – Hao He
    Commented Mar 20, 2023 at 1:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .