Distribution of Submatrix of Orthogonal Matrix

Question

Let $O$ be a matrix sampled from the Haar measure on $O(n)$. Let $X$ be the upperleft $k\times k$ submatrix of $O$.

In a physics research project I am interested in the distribution of $X$, say $\rho(X)$. What I was able to prove for $k=1$ and $k=2$ is that:

$k=1$: $\rho(X)\propto(1-X^2)^{(n-3)/2}$.

$k=2$: $\rho(X)\propto (1-\text{Tr}(XX^T)+\det(XX^T))^{(n-5)/2}$.

It did not escape my attention that these expressions are in fact closely related to the characteristic polynomial $p(x)$ of $XX^T$. In both cases, the distribution $\rho(X)$ is a function of $p(1)$.

Given the simplicity of the result, I believe that this result is known to the math community for a long time; however, I can not find any literature that contains the result. I would be grateful if someone can point me to some existing literature or enlighten me on the significance of the characteristic polynomial.

Answer 1

The probability distribution of $X$ was calculated in Random-matrix theory of thermal conduction in superconducting quantum dots. In the context of that physics problem, the $k\times k$ upper-left submatrix $X$ of an $n\times n$ orthogonal matrix $O$ is the reflection matrix of a superconducting quantum dot with $k$ modes in one opening and $n-k$ modes in the other opening. Such a system is described by the circular real ensemble (CRE) of random-matrix theory.

The probability distribution of $X$ is conveniently described in the singular value decomposition $X=O_1 {\rm diag}\,(x_1,x_2,\ldots x_k)O_2$, where $x_i\geq 0$ are the singular values and $O_1,O_2$ are orthogonal matrices.

The matrices $O_1,O_2$ are independently drawn from $O(k)$ with the Haar measure. The singular values $x_i$ of $X$ are related to the transmission probabilities $T_i$ by $T_i=1-x_i^2$. For $N=\min(k,n-k)$ there is a set $\{T_1, T_2,\ldots T_N\}$ of nonzero transmission probabilities that have the probability distribution $$P(\{T_1,T_2,\ldots T_N\})\propto \prod_i T_i^{|n/2-k|-1/2}(1-T_i)^{-1/2}\prod_{j<\ell}|T_\ell-T_j|.$$ _{See equation (5) in the cited paper, with $\beta=1$ and $\gamma=-1$.}

Equivalently, in terms of the $x_i$'s themselves, there is a set $\{x_1, x_2,\ldots x_N\}$ of singular values that are strictly smaller than unity in absolute value, with probability distribution $$P(\{x_1,x_2,\ldots x_N\})\propto \prod_i (1-x_i^2)^{|n/2-k|-1/2}\prod_{j<\ell}|x_\ell^2-x_j^2|.$$

The probability distribution of the matrix elements of the matrix $X$ can indeed be written (for $k<n/2$) as $P(\{X_{ij}\})\propto {\rm Det}\,(1-XX^\top)^{n/2-k-1/2}$, as in the OP. Equivalently, in terms of the symmetric matrix $H=XX^\top$ one has $P(\{H_{ij}\})\propto {\rm Det}\,H^{-1/2}\,{\rm Det}\,(1-H)^{n/2-k-1/2}$.

The OP also asks for the significance of the appearance of the characteristic polynomial ${\rm Det}(1-XX^\top)$. This is easiest to understand for $k>n/2$, when $XX^T$ has $k-n/2$ eigenvalues pinned at unity. These repel the $n/2$ other eigenvalues with a term $\prod_{i=1}^{n/2}(1-x_i^2)^{k-n/2}$.