# Action of symmetric matrices under $\mathrm{O}(n)$

$$\DeclareMathOperator\Sym{Sym}\DeclareMathOperator\O{O}\DeclareMathOperator\GL{GL}$$Let $$k$$ be an algebraically closed field of characteristic 0 (it can even be $$\mathbb{C}$$ if you like), and let $$n\in\mathbb{N}$$. Let $$\Sym(n)$$ denote the space of $$n\times n$$ symmetric matrices with entries in $$k$$. Let $$\O(n)\subseteq \GL(n)$$ denote the orthogonal group, that is all $$n\times n$$ matrices $$A$$ with entries in $$k$$ such that $$AA^t=A^tA=I_n$$.

Then $$\O(n)$$ acts on $$\Sym(n)$$ by conjugation. I would like nice representatives for the orbits of this action.

Let $$B\in \Sym(n)$$. If $$B$$ is diagonalizable, then it is a theorem that it can be diagonalized by an orthogonal matrix, so all the 'semisimple orbits' have a diagonal matrix as a representative.

So my question is about when $$B$$ is not diagonalizable — what is a 'nice' form we can put it in under orthogonal conjugation? (For an example of a non-diagonalizable symmetric matrix, see https://math.stackexchange.com/questions/1658393/why-a-complex-symmetric-matrix-is-not-diagonalizible.)

Seeking explanations or references. Thank you!

The adjoint action of $$SO(n,\mathbb C)$$ on $$\mathrm{Sym}_0(n,\mathbb C)$$ (we can replace $$O(n,\mathbb C)$$ by $$SO(n,\mathbb C)$$, and $$\mathrm{Sym}(n,\mathbb C)$$ by the subspace of traceless matrices, since the action splits off the scalar multiples of the identity) is the isotropy representation of the complex symmetric space $$SL(n,\mathbb C)/SO(n,\mathbb C)$$. Kostant and Rallis (1971) investigated such representations. This was later generalized by Vinberg (1976) to the case of a $$\theta$$-group (the $$\mathfrak g_0$$-action on $$\mathfrak g_1$$, where we consider the decomposition of a complex Lie algebra $$\mathfrak g=\mathfrak g_0 +\mathfrak g_1+\cdots + \mathfrak g_m$$ into eigenspaces of an automorphism $$\theta$$ of order $$m$$ ($$1$$-parameter group of automorphisms in case $$m=\infty$$).

Among other things, they found that for such a representation $$(G_0,V)$$:

1. There is a Jordan decomposition: every $$x\in V$$ can be uniquely written as $$x_s+ x_n$$, where $$x_s$$ is a semisimple element (this means that $$ad_{x_s}$$ is a semisimple transformation) , $$x_n$$ is a nilpotent element (this means that $$\mathrm{ad}_{x_n}$$ is a nilpotent transformation), and they commute.
2. A maximal Abelian subspace $$c$$ of $$V$$ consisting of semisimple elements is called a Cartan subspace. Any two Cartan subspaces are $$G_0$$-conjugate.
3. Semisimple elements are characterized by the fact that their $$G^0$$-orbits are closed, or they lie in a Cartan subspace, or they are $$G^0$$-conjugate to an element in a given Cartan subspace. In your example, a Cartan subspace is given by the diagonal matrices, and the semisimple elements are exactly the diagonalizable symmetric matrices.
4. Nilpotent elements are characterized by the fact that $$0$$ lies in the closure of their orbits. Nilpotent elements of course come in $$G^0$$-orbits, and there are only finitely many nilpotent orbits. In the case of a complex symmetric space $$\mathfrak g=\mathfrak k + \mathfrak p$$ (under $$\theta$$), the Kostant-Sekiguchi correspondence establishes a bijection between the set of nilpotent $$K$$-orbits on $$\mathfrak p$$ and the set of nilpotent $$G_{\mathbb R}$$-orbits on $$\mathfrak g_{\mathbb R}$$, where $$\mathfrak g_{\mathbb R}$$ is the noncompact real form of $$\mathfrak g$$ whose Cartan involution induces $$\theta$$ on $$\mathfrak g$$. In turn, the nilpotent $$G_{\mathbb R}$$-orbits on $$\mathfrak g_{\mathbb R}$$ are parametrized by partitions.

More explicitly, in our case we have the symmetric space $$\mathfrak{sl}(n,\mathbb C)=\mathfrak{so}(n,\mathbb C)+\mathrm{Sym}_0(n,\mathbb C)$$ and the nilpotent $$SO(n,\mathbb C)$$-orbits in $$\mathrm{Sym}(n,\mathbb C)$$ are in bijection with the nilpotent $$SL(n,\mathbb R)$$-orbits in $$\mathfrak{sl}(n,\mathbb R)$$. The latter can be understood in terms of the Jordan canonical form. As an explicit example of the Kostant-Sekiguchi correspondence, the nilpositive element $$\begin{pmatrix}0&1\\0&0\end{pmatrix}$$ of an $$\mathfrak{sl}_2$$-triple in $$\mathfrak{sl}(2,\mathbb R)$$ corresponds under the Cayley transform to the nilpositive element $$\frac12\begin{pmatrix}i&1\\1&-i\end{pmatrix}$$ of a normal $$\mathfrak{sl}_2$$-triple in $$\mathfrak{sl}(2,\mathbb C)$$; that is essentially the example given by @JohannesEbert. One can work the details of the correspondence more generally, see e.g. the book
https://www.amazon.com/Nilpotent-Orbits-Semisimple-Lie-Algebra/dp/0534188346

• Thank you, this is what I needed! Feb 8 at 16:25

Let me give a more concrete and now (hopefully) correct answer. I've typed up the details mostly for my own education.

I'll work over $$\mathbb{C}$$, but the argument generalizes directly to an algebraically closed field of characteristic $$0$$. Let us say that a subspace $$V\subset \mathbb{C}^n$$ is orthogonally complemented when the orthogonal annihilator $$V^\perp = \{u\in \mathbb{C}^n \mid u^t V=0 \}$$ is disjoint from the original subspace, $$V^\perp \cap V = \{0\}$$, in which case $$\mathbb{C}^n \cong V\oplus V^\perp$$. The problem with the standard argument about the diagonalizability of a symmetric matrix, as pointed out in the comments, is that a complex eigenvector $$u$$ is does not span an orthogonally complemented invariant subspace when the eigenvector is null/isotropic, $$u^t u = 0$$.

The correct result is the following. For convenience, let $$D_m$$ denote the $$m\times m$$ matrix with $$1$$s on the main anti-diagonal and zeros everywhere else, $$(D_m)_{j,k} = \delta_{j+k,m+1}$$.
Theorem. Given a complex symmetric matrix $$A = A^t$$, there exists a basis $$n_j$$ in which $$A$$ is in Jordan canonical form and the matrix of inner products $$P=(n_j^t n_k)_{j,k}$$ is block diagonal. For each Jordan block of size $$m$$, the corresponding diagonal block in $$P$$ is equal to $$D_m$$.

As one can see, the basis $$n_j$$ is not orthonormal, except for when all the Jordan blocks are trivial, the diagonalizable case. However, one can easily choose a canonical orthonormal basis within each of the $$D_m$$ normalized blocks, the only price being an exchange of the usual form of the Jordan blocks to a different canonical form. The new orthonormal basis will give you the orthogonal transformation needed to bring $$A$$ to this new canonical form.

The proof mimics the standard construction of the Jordan canonical form, with the exception that one needs to keep track of the inner products between the Jordan basis within each block. A complete proof can be reconstructed from the following lemmas.

Lemma 1. If $$V\subset \mathbb{C}^n$$ is an orthogonally complemented $$A$$-invariant subspace, then $$A$$ is block diagonal with respect to the decomposition $$\mathbb{C}^n = V \oplus V^\perp$$.
Proof. Pick an orthonormal basis adapted to the decomposition. $$A$$ is symmetric in this basis and $$V$$ is $$A$$-invariant. Hence both off-diagonal blocks must vanish. $$\square$$ This is the standard argument.

Lemma 2. If $$\lambda$$ be an eigenvalue of $$A$$, then the generalized eigensubspace $$V_\lambda \subset \mathbb{C}^n$$ is orthogonally complemented, where $$V_\lambda$$ contains all vectors annihilated by some power of $$(A-\lambda)$$. Proof. Let $$\mu\ne \lambda$$ be any other eigenvalue of $$A$$. Consider generalized eigenvectors $$(A-\lambda)^k v = 0$$ and $$(A-\mu)^l w = 0$$. Then, rewriting $$\mu-\lambda = (A-\lambda) + (\mu-A)$$, we find the identity $$(\mu-\lambda)^{m} w^t v = \sum_{j=0}^m \binom{m}{j} [(\mu-A)^{j} w]^t (A-\lambda)^{m-j} v.$$ For sufficiently large $$m\ge k+l-1$$, all terms on the right-hand side vanish, implying $$w^t v = 0$$. Hence, $$V_\lambda$$ is orthogonal to the subspace $$W$$ spanned by the generalized eigenvectors for all other eigenvalues of $$A$$, while $$\mathbb{C}^n \cong V_\lambda \oplus W$$. $$\square$$ When $$k=l=1$$, this is also part of the standard argument.

Lemma 3. Suppose that $$A$$ is also nilpotent (it only has the $$0$$ eigenvalue). We can arrange the Jordan form of $$A$$ so that if $$V\subset \mathbb{C}^n$$ is an $$A$$-invariant subspace corresponding to a single Jordan block, then $$V$$ is orthogonally complemented. Moreover, the Jordan basis can be chosen to have the anti-diagonal inner product structure.
Proof. Every Jordan block of size $$m$$ is generated by a generalized eigenvector $$v_m$$ such that $$A^m v_m = 0$$, but $$v_{j\ge 1} := A^{m-j} v_m \ne 0$$. Let us assume that $$m$$ is maximal, that is $$A^m = 0$$. After an orthogonal block diagonalization via Lemma 1, using the same argument recursively covers all Jordan block sizes. Either (a) $$v_1^t v_m \ne 0$$ or (b) $$v_1^t v_m = 0$$.
We first reduce case (b) to (a). Pick any vector $$w_m$$ such that $$v_1^t w_m = 1$$, which is always possible, and define $$w_j = A^{m-j} w_m$$, so that $$v_j^t w_{m+1-j} = v_m^t A^{m-1} w_m = 1$$. In particular, this shows that $$w_1 \ne 0$$, so that $$w_1,\ldots, w_m$$ span the subspace $$W$$ of a Jordan block of the same size. Again, either (a') $$w_1^t w_m \ne 0$$ or (b') $$w_1^t w_m = 0$$. In the first case, we simply replace $$v_m \to w_m$$ and continue the proof. In the second case, we take for instance $$v_m\to v_m+w_m$$.
Now we proceed under assumption (a). We can normalize $$v_m$$, and hence $$v_1 = A^{m-1} v_m$$ such that $$v_1^t v_m = 1$$ (uses algebraic closure of $$\mathbb{C}$$). Note that $$v_j^t v_k = (A^{m-j} v_m)^t (A^{m-k} v_m) = v_m^t A^{2m-j-k} v_m = c_{j+k}$$. For $$j+k, $$c_{j+k}=0$$ since $$A^m=0$$, while, for $$j+k=m+1$$, $$c_{j+k}=1$$ by the earlier normalization. In general $$c_{j+k}\ne 0$$ for $$j+k>m+1$$, but we can use the freedom in choosing the Jordan basis, $$v_m \to v_m + \sum_{j=1}^{m-1} b_j v_j$$ and choose the $$b_j$$ to set these $$c_{j+k}=0$$. Clearly, since now $$v_j^t v_k = \delta_{j+k,m+1}$$, the subspace $$V$$ spanned by the Jordan basis $$v_1,\ldots, v_m$$ is orthogonally complemented. $$\square$$

I adapted the proof of Lemma 3 from the proof or a related, but slightly different, Theorem 5.1.1 (which deals with matrices Hermitian with respect to an indefinite inner product) in

Gohberg, Israel; Lancaster, Peter; Rodman, Leiba, Indefinite linear algebra and applications, Basel: Birkhäuser (ISBN 3-7643-7349-0/pbk). xii, 357 p. (2005). ZBL1084.15005.

The notes in §5.19 of this reference give historical references for this result.

OLD INCORRECT ARGUMENT: The usual proof of the theorem you quote also proves that, at least for algebraically closed fields, a symmetric matrix is always diagonalizable: Given an eigenvector, the orthogonal complement is an invariant subspace, hence the matrix is block diagonalizable by an orthogonal transformation, splitting off the 1-dimensional subspace spanned by the eigenvector (does not work when the norm of the eigenvector is $$0$$, see comments) . Whenever the characteristic polynomial of the matrix has a root, there exists an eigenvector. And the algebraic closedness of the field ensures that a root always exists. Proceeding by induction, the matrix is brought to diagonal form by a sequence of orthogonal transformations.

• That's not true: over the complex numbers, there are nonzero vectors which are isotropic, e.g. (1,i). So the argument fails, for example for the matrix (1,i,i,-1 ) over the complex numbers, which is symmetric and nilpotent, and not diagonalizable. Feb 5 at 0:00
• Hi Igor, Johannes is correct. See this post as well: math.stackexchange.com/questions/1658393/… Feb 5 at 8:04
• @JohannesEbert Ah, yes, good point! So the argument fails because the orthogonal annihilator of a subspace is not disjoint from it. The same problem occurs for real inner products with indefinite signature. So by the Jordan-Chevalley decomposition, the correct answer comes down to classifying the orbits of symmetric nilpotent matrices. Feb 5 at 11:43
• @freeRmodule I've updated my answer to be (hopefully) correct and quite concrete. Feb 8 at 0:50
• @freeRmodule Yes, I've corrected Lemma 2 and its proof. I on the other hand, I find it difficult to see what the Kostant-Sekiguchi correspondence says about the orthogonality properties of Jordan blocks, at least from what's in Gorodski's answer. But what you say actually also agrees with my statement of Lemma 3, by spotting a simplification according to which case (b) reduces to case (a), which I failed to notice. Since the two Jordan blocks are identical, they can be mixed into two new, orthogonal blocks. Feb 9 at 12:45