Frobenius norm of the principal submatrix of a uniformly distributed random orthonormal matrix Suppose that we have a uniformly distributed $d\times d$ random orthonormal matrix $\mathbf{X}$. Here "uniform" is defined in the sense of Haar measure, i.e., the distribution does not change up to any rotation of basis (e.g., multiplication with any arbitrary orthonormal matrix).
Let $\mathbf{Z}$ be the upper-left $k\times k$ principal submatrix of $\mathbf{X}$. What is the distribution of $\| \mathbf{Z} \|_F^2$, e.g., its probability density function and cumulative distribution function? (We may be also interested in their asymptotic versions.)
This also relates to the so-called multivariate beta distribution. Let $\mathbf{Y}\in \mathbb{R}^{d\times k}$ be a random matrix with i.i.d. standard Gaussian entires, and $\mathbf{Y} = \mathbf{Q}\mathbf{R}$ be its QR decomposition. Let $\mathbf{Q}'$ be the top $k\times k$ submatrix of $\mathbf{Q}$. Then $\mathbf{Q}'$ follows the multivariate beta distribution. An equivalent question is as follows. What is the (asymptotic) distribution of $\|\mathbf{Q}'\|_F^2$?
 A: This problem has been studied in the physics literature as the distribution of the thermal conductance of a superconducting quantum dot. Let me explain the relationship: The $d\times d$ orthogonal matrix $X$ corresponds to the scattering matrix $S$, the $k\times k$ upper-left principal submatrix $Z$ corresponds to the reflection matrix $r$, the Frobenius norm squared $||Z||_F^2$ is related to the dimensionless thermal conductance $G$ by
$$G=k-{\rm tr}\,rr^{\rm T}=k-||Z||_F^2.$$
(Superscript T is the transpose and tr is trace.)
The distribution of $G$, and hence of the desired Frobenius norm, follows from the distribution of the socalled transmission eigenvalues $T_n=1-R_n$, with $R_n$ an eigenvalue of $ZZ^{\rm T}$. The joint probability distribution of the $R_n$'s is known exactly for any $k$ and $d=k+k'$. In the case $k\leq k'$ it is given by
$$P(R_1,R_2,\ldots R_k)=C \sqrt{\prod_{n=1}^k\frac{(1-R_n)^{k'-k}}{(1-R_n)R_n}}\;\prod_{i=1}^j\prod_{j=1}^k |R_i-R_j|,$$
with $C$ a normalization constant and all $R_n\in(0,1)$. If $k>k'$ the above result still holds with $k$ and $k'$ interchanged, and then there are an additional set of $k-k'$ of the $R_n$'s pinned at unity.
The desired probability distribution of
$$||Z||_F^2=\sum_{n=1}^k R_n$$
now follows upon integration. If $d\gg k$ an approximate result follows by taking the matrix elements of $Z$ to be independent Gaussians, of zero mean and variance $1/d$, resulting in a chi-squared distribution with $k^2$ degrees of freedom.
If knowledge of the first two moments is sufficient, then there are exact results for any $k,d$,
$$E(||Z||_F^2)=k^2/d,\;\;{\rm Var}\,(||Z||_F^2)=\frac{2k^2(d-k)^2}{d^2(d-1)(d+2)}$$
I note that there are similar results when $X$ is complex unitary, rather than real orthogonal, reviewed here.
