Fundamental domain for two Grassmannians

Question

Let $\pi_1, \pi_2$ be two $k$-dimensional subspaces of $\mathbb R^n$. Using elements of the orthogonal group $O(n)$, how much can we simplify $\pi_1, \pi_2$? Certainly there always exists $A \in O(n)$ such that $A \cdot \pi_1$ is the span of the first $k$ canonical bases, but this doesn't say anything about $A \cdot \pi_2$.

More precisely, I'd like to know if there is a standard, well-known fundamental domain of the $O(n)$-action on $Gr(k, n) \times Gr(k, n)$, given by $A \cdot (\pi_1, \pi_2) = (A \pi_1, A \pi_2)$. Since $\dim Gr(k, n) = k(n-k)$ and $\dim O(n) = n(n-1)/2$, the quotient space should have the following dimension: $$\dim \frac{Gr(k, n) \times Gr(k, n)}{O(n)} = 2k(n-k) - \frac12 n(n-1) = -2 \left( k^2 -nk + \frac 14 (n^2 - n) \right)$$ This is a quadratic expression in $k$, and it is a positive number when $n - \sqrt n < 2k < n + \sqrt n$. (For example see this) Outside this range, the quotient space should be a discrete collection of points. Within the range, $O(n)$ shouldn't be able to exhaust the degrees of freedom of $Gr(k, n) \times Gr(k, n)$, and thus the quotient space should have extra degrees of freedom. As a simple example, $(k, n) = (2, 4)$ will produce 2 degrees of freedom and $(k, n) = (1, 4)$ will produce 0 degrees of freedom.

Answer 1

Your argument about the dimension of the quotient doesn't take into account that there may be elements of the orthogonal group that don't do anything to the pair $(\pi_1,\pi_2)$. For example, if $2k<n-1$, then the span of the two planes has codimension at least $2$ in $\mathbb{R}^n$, so there will be rotations that are the identity on both $\pi_1$ and $\pi_2$, and you have to subtract that stabilizer as well.

More generally, here is what you can say: Define a quadratic form $Q$ on $\pi_1$ by letting $Q(v) = |v'|^2$ where $v'$ is the orthogonal projection of $v\in \pi_1\subset\mathbb{R}^n$ onto $\pi_2$. Since $|v'|^2\le |v|^2$ with equality if and only if $v$ lies in $\pi_2$, we see that the eigenvalues of $Q(v)$ with respect to the 'natural' norm $|v|^2$ are all less than or equal to $1$. Let those eigenvalues be $1\ge \cos(\theta_1)^2\ge \cos(\theta_2)^2\ge\cdots\ge\cos(\theta_k)^2\ge 0$, where $\theta_i\in[0,\pi/2]$. Then there will exist an orthonormal basis $e_1,\ldots,e_k$ of $\pi_1$ and an orthonormal basis $f_1,\ldots,f_k$ of $\pi_2$ such that $e_i' = \cos\theta_i\, f_i$.

In fact, it's not hard to see that the quantity $\theta = (\theta_1,\theta_2,\ldots,\theta_k)$, where $0\le\theta_1\le\theta_2\le\cdots\le\theta_k\le \pi/2$, completely determines the pair $(\pi_1,\pi_2)$ up to $\mathrm{O}(n)$ equivalence. Moreover, the sequence $\theta$ has to start with at least $2k{-}n$ zeroes (since the intersection of $\pi_1$ and $\pi_2$ has to have dimension at least $2k{-}n$). Beyond this, there is no restriction on the $\theta_i$, so the moduli space is a $j$-simplex where $j= k - \max(0,2k{-}n) = \min(k,n{-}k)$.

N.B.: I feel that I would be remiss in not mentioning that this is a very special case of the 'two-point' problem for Riemannian symmetric spaces. The Grassmannian of $k$-planes in $\mathbb{R}^n$ is, of course, a Riemannian symmetric space of rank $\min(k,n{-}k)$, and the above description generalizes to Riemannian symmetric spaces of rank $r$, i.e., that the moduli space of pairs of points in a Riemannian symmetric space $M=U/K$ of rank $r$ is naturally an $r$-dimensional polyhedron. If you want to know more about the general case, I suggest consulting a good text on symmetric spaces (cf. Helgason) that describes the action of the reduced Weyl group.