Distribution of inverse of a random matrix

Question

I got stuck into a problem and couldn't find its satisfactory answer anywhere.

My question is simple. Suppose I have a fat random matrix (i,e., $R$ has dimensions $k\times d$ where $k<d$) whose elements are chosen from a i.i.d. standard normal distribution $N(0,1)$.
Suppose I find its pseudo-inverse, given by: $R^+ = (R' R)^{-1} R'$.

1. Will this pseudo-inverse matrix still remain random ?
2. If yes, will it contain elements distributed with normal distribution?
3. If yes, what would be the mean and variance of this this normal distribution?

I am asking these questions because I have experimented with a lot of random matrices (with elements distributed with $N(0,1)$. When in plot a histogram of pseudo inverse elements, it comes a normal distribution with mean $= 0$ and variance $= 1/(\text{variance of }R \times d^2)$ ; where $d$ are the columns in $R.$)

I have tried to find the PDF using Jacobian transform but i could not figure out how will it shape up the variance.

I would be thankful if you could guide me or clarify my problem.

Answer 1

It is true under the assumption $k,d\to+\infty$ while $k/d\to 0$, in the sense that $R^T\in R^{d\times k}$ has obviously iid $N(0,1)$ entries, and satisfies $$ \|\sqrt d R^+ - R^T/\sqrt d\|_{op} \to^P 0. $$ First let us recall some well known concentration inequalities for the smallest and largest singular values of $R$, namely $$ P( \sqrt{d} - \sqrt{k} - t \le s_{\min}(R) \le s_{\max}(R) = \|R\|_{op} \le \sqrt d + \sqrt k + t) \le 2e^{-t^2/2}. $$ if $k/d\to 0$, one can for instance use $t=\sqrt{\log(d)}$ to obtain that $s_{\min}(R)/\sqrt d\to^P 1$ and similarly $\|R\|_{op}/\sqrt d\to ^P 1$.

Let us now explain why $\|\sqrt d R^+ - R^T/\sqrt d\|_{op} \to^P 0$. Consider the SVD $R=UDV^T$. Then the pseudo-inverse is $R^+ = VD^{=1} U^T$ and \begin{align*} \|\sqrt d R^+ - R^T/\sqrt d\|_{op} &=\|U(\sqrt d D^{-1} - d^{-1/2} D)V\|_{op} \\&=\|\sqrt d D^{-1} - d^{-1/2} D\|_{op} \\&\le \|\sqrt d D^{-1} -I_k\|_{op} + \|I_k - d^{-1/2} D\|_{op}. \end{align*} If $k/d\to 0$, the right-hand side converges to 0 in an event of probability at least $1-2/p$ by taking $t=\sqrt{\log d}$ in the above concentration inequality.

Answer 2

This is not a full answer, but firstly for 1, yes, of course. What's more, the elements of $R^+$ are independent of each other, because the elements of $R$ are independent of each other, and the pseudo-inverse preserves matrix rank.

For 3, $E[R^+_{ij}]=0$, due to symmetry, and $E[(R^+_{ij})^2]=\frac{1}{d(d-k-1)}$, as can be shown from an analysis in this paper. If you look into that paper, keep in mind that the singular inverse-Wishart distributed matrix would be $R^{+\top}R^+$, so the diagonal elements of the mean of that matrix are simply $k$ times the variance of each element, i.e. $E[R^{+\top}R^+]=kE[(R^+_{ij})^2]I_d$. This is because the diagonal elements in the matrix product given in section 4.1 $R^{+\top}R^+$ each sum $k$ squared (independent) elements of $R^+$. Plugging in the derived equation from the paper, which in this post's notation would read $E[R^{+\top}R^+]=\frac{k}{d(d-k-1)}I_d$, gives us our answer. One could also get here via a similar path using the mean of the non-singular Wishart distributed random matrix, $E[R^+R^{+\top}]$.

As for 2, while each element of $R^+$ is generally not Gaussian (consider k=d=1), I would certainly believe they approach Gaussianity as the dimensions increase, but I don't know anything about that.