perturbation of Invariant subspaces Let $A,B$ be matrices in $GL(n,\mathbb{R})$ sufficiently close in the usual metric on matrices. Suppose $A$, resp., $B$ stabilizes a $k$-dimensional subspace $U$, resp., $V$ of $\mathbb{R}^n$, where $0 < k < $ dim $V$. Is there any topological relationship between $U$ and $V$? Is there any metric on the set of $k$-dimensional spaces, and if Yes can it be said that $U$ and $V$ are close in that metric?
 A: One "natural" metric on the $k$-dimensional subspaces is the Hausdorff metric on their intersections with the unit ball.
Consider the case $A = I$.  Every $k$-dimensional subspace is invariant under $I$.  Let $C$ be a matrix with all eigenvalues distinct, and consider $B = I + t C$ for small $t$.  The subspaces invariant under $B$ are the same as the subspaces invariant under $C$, and there are only $n \choose k$ of them that are
$k$-dimensional.  In particular, you can choose a $U$ that is not one of these,
and no $V$ will be close to $U$ as $t \to 0$.
A: You should read Gohberg, Lancaster, Rodman, Invariant subspaces with Applications, Part III. Very detailed results are contained there.
Another good reference, more applied/numerical, is Stewart, Sun, Matrix perturbation theory, whose Part 2 is on invariant subspaces. It contains several closed-form bounds for the distance.
Quick TL;DR version (but go check the book anyway, there is much more there): the (semi-)natural metric to use is $\|P_U-P_V\|$, where $P_Z$ is the orthogonal projector on $Z$ and the norm is any matrix norm, usually the Euclidean one. If the eigenvalues associated to $U$ are distinct from those not associated to it, then the invariant subspace is Lipschitz stable under small perturbations. On the other hand, if your invariant subspace 'sees' an eigenvalue of algebraic multiplicity $k$ of $A$ with strictly smaller multiplicity $0<k'<k$, then things become trickier (as you may expect: e.g., perturb $A=I$).
If $A$ is Hermitian, then the Lipschitz constant is the inverse of the minimum difference between these two sets of eigenvalues; if $A$ is nonsymmetric, then this Lipschitz constant (known as separation) can be arbitrarily worse than the spectral gap (e.g., take $\begin{bmatrix}1 & k\\ 0 & 2\end{bmatrix}$ for large $k$; its eigenvectors are very close, although the eigenvalues are separated): there is an expression for it as the solution of a matrix equation but it's often unwieldy.
