# A question on linear groups

Asking this question I have made a mistake joining my main question with two simple ones, so it hasn't received enough attention, however there was a partial answer, which was not elaborated, and now I am completely lost =)

My original question goes as follows (with a little editing).

Q1: Suppose we have a linear operator $T:\mathbb{R}^n\to \mathbb{R}^n$, $n>2$, which is NOT a constant times an isometry. Let $G$ be the subgroup of $GL_n(\mathbb{R})$, generated by the orthogonal group together with $T$. I need to prove (or refute) that $G$ acts "bitransitively" on $\mathbb{R}^n$, in the sense that it can map any pair of non-colinear vectors into any other pair of non-colinear vectors. I also need a similar result for the complex case.

A comment that I received states that this is true, due the following:

Q2: $\mathbb{R}^*O_n(\mathbb{R})$ is a maximal subgroup in $GL_n(\mathbb{R})$ (and analogously for the complex case).

I don't really see neither why these statements hold, nor how to derive my question from them, except of the case when $\det T=\pm1$. I have found two papers that seem to have proofs of these statements, but I don't really have a background in algebra, so I can't even understand if this is the case (I am not sure if the Euclidean spaces fit into the scope of these papers). Furthermore, I am convinced that there should be some geometry behind this issue, and there is no chance that I can retrieve the geometric intuition from theses papers, again since my background doesn't allow me to read them. And as I've mentioned before, I don't see how to derive Q1 from Q2 in general case.

I hope you can help me with this predicament.

I will also add another related (vaguely stated) question.

Q3: What can be said about a subgroup $G$ of $GL_n(\mathbb{R})$, if it can map any $k$-tuple of linearly independent vectors into any other $k$-tuple, for some $k\in\overline{1,n-1}$?

• For $n=2$ it is wrong. Take $T$ of determinant one. – Wilberd van der Kallen Jun 30 '16 at 7:25
• Q2 holds true: indeed here determinant does not matter since the subgroup already contains all homotheties and contains elements of all determinants. After modding out by homotheties, the question boils down to whether $PO(n)(R)$ is maximal in $PGLn(R)$, and this is true and follows from the fact that $SO(n)(R)$ is maximal in $SL_n(R)$. Here $R$ are the reals. – YCor Jun 30 '16 at 7:50
• @WilberdvanderKallen I first misread your comment and I refer to Q2 – YCor Jun 30 '16 at 7:50
• I should have said that Q1 is wrong for $n=2$. – Wilberd van der Kallen Jun 30 '16 at 7:54
• Indeed. Still for all $n$ it's true that a subgroup containing the orthogonal group and not contained in the group of similarities, will contain $SL_n(R)$. Thus Q1 is true for $n\ge 3$. – YCor Jun 30 '16 at 8:12

Here's a complete proof of:

Every subgroup of $\mathrm{GL}_n(\mathbf{R})$ containing $\mathrm{SO}_n(\mathbf{R})$ is either contained in the group of similarities $\mathbf{R}^*\mathrm{O}_n(\mathbf{R})$, or contains $\mathrm{SL}_n(\mathbf{R})$.

(This is equivalent to the statement that for every $t\in\mathrm{GL}_n(\mathbf{R})\smallsetminus \mathbf{R}^*\mathrm{O}(n)$, the subgroup generated by $\mathrm{SO}_n(\mathbf{R})\cup\{t\}$ contains $\mathrm{SL}_n(\mathbf{R})$. The answers to Q2 for $n\ge 2$ and to Q1 for $n\ge 3$ immediately follow.)

Let $H$ be a subgroup of $\mathrm{GL}_n(\mathbf{R})$. Let $\mathfrak{h}$ be the set of $x\in\mathfrak{gl}_n(\mathbf{R})$ such that there exists a $C^1$ map $u:\mathbf{R}\to\mathrm{GL}_n(\mathbf{R})$ such that $u(0)=I_n$ (the identity matrix), $u'(0)=x$, and $u$ takes values in $H$. Then $\mathfrak{h}$ is a linear subspace. Indeed, it's clearly stable under scalar multiplication, and if $u,v$ are two such maps with $u'(0)=x$, $v'(0)=y$, then defining $w(t)=u(t)v(t)$ we have $w'(0)=x+y$.

Now assume that $H$ contains $\mathrm{SO}_n(\mathbf{R})$.

First assume that $H$ is contained in $\mathrm{SL}_n(\mathbf{R})$; it follows that $\mathfrak{h}$ is contained in $\mathfrak{sl}_n(\mathbf{R})$.

Then $\mathfrak{h}$ is invariant under the action by conjugation of $\mathrm{SO}_n(\mathbf{R})$. Now (as mentioned in the post by Venkataramana), we can decompose $\mathfrak{sl}_n(\mathbf{R})$ as a $\mathrm{SO}_n(\mathbf{R})$-representation, namely $\mathfrak{sl}_n(\mathbf{R}) = \mathfrak{so}_n(\mathbf{R})\oplus\mathfrak{p}$, where $\mathfrak{p}$ is the set of symmetric matrices with trace zero. Then for all $n \ge 2$ (including $n=2$), $\mathfrak{p}$ is irreducible (see a proof below); it follows that the subspaces of $\mathfrak{sl}_n(\mathbf{R})$ containing $\mathfrak{so}_n(\mathbf{R})$ and invariant under conjugation by $\mathrm{SO}_n(\mathbf{R})$ are $\mathfrak{so}_n(\mathbf{R})$ and $\mathfrak{so}_n(\mathbf{R})\oplus\mathfrak{p}=\mathfrak{sl}_n(\mathbf{R})$.

If $H$ is contained in $\mathrm{SO}_n(\mathbf{R})$ then we are in one of the first cases. Conversely, if $H$ is not contained in $\mathrm{SO}_n(\mathbf{R})$ then since the latter is the stabilizer of $\mathfrak{so}_n(\mathbf{R})$ in $\mathrm{SL}_n(\mathbf{R})$ (see a proof below), we deduce that $\mathfrak{h}$ contains a conjugate of $\mathfrak{so}_n(\mathbf{R})$ distinct from $\mathfrak{so}_n$. Hence we are in the second case, that is, $\mathfrak{h}=\mathfrak{sl}_n(\mathbf{R})$. In this case we can pick a basis $(e_1,\dots,e_m)$ of $\mathfrak{sl}_n(\mathbf{R})$ ($m=n^2-1$) and functions $u_1,\dots,u_m:\mathbf{R}\to H$ as in the definition of $\mathfrak{h}$ with $u'_i(0)=e_i$. It follow that the differential of the function $(u_1,\dots,u_m):\mathbf{R}^m\to H\subset\mathrm{SL}_n(\mathbf{R})$ at zero is surjective. Hence its image contains a neighborhood of $I_n$ in $\mathrm{SL}_n(\mathbf{R})$. Thus $H$ is open in $\mathrm{SL}_n(\mathbf{R})$ and hence by connectedness of the latter, is equal to $\mathrm{SL}_n(\mathbf{R})$.

Now remove the assumption that $H$ is in $\mathrm{SL}_n(\mathbf{R})$. If $H\cap\mathrm{SL}_n(\mathbf{R})=\mathrm{SO}_n(\mathbf{R})$, then $H$ is contained in the normalizer $\mathbf{R}^*\mathrm{O}_n(\mathbf{R})$ of $\mathrm{SO}_n(\mathbf{R})$. Otherwise, we deduce from the previous case that $H\cap\mathrm{SL}_n(\mathbf{R})$ equals $\mathrm{SL}_n(\mathbf{R})$, that is to say, $H$ contains $\mathrm{SL}_n(\mathbf{R})$. This finishes the proof.

Proof that the action of $\mathrm{SO}_n(\mathbf{R})$ on $\mathfrak{p}$ is irreducible for $n\ge 2$. First, consider the action of the subgroup of signed permutation matrices with determinant 1 on the subspace $D\subset\mathfrak{p}$ of diagonal matrices with trace zero. It factors through the action of permutation matrices (not only even ones, using $n\ge 2$) on the set of $n$-vectors with sum zero (by permutation of coordinates), and this action is well-known and easily checked to be irreducible.

Now let $V$ be a nonzero $\mathrm{SO}_n(\mathbf{R})$-invariant subspace of $\mathfrak{p}$. Picking a nonzero element in $V$, it has a $\mathrm{SO}_n(\mathbf{R})$-conjugate that is diagonal. Therefore $V\cap D\neq 0$, and by the previous irreducibility fact, $D\subset V$. Now any element of $\mathfrak{p}$ has a $\mathrm{SO}_n(\mathbf{R})$-conjugate that is diagonal, so $V=\mathfrak{p}$. $\Box$

Proof that the stabilizer $N$ of $\mathfrak{so}_n(\mathbf{R})$ in $\mathrm{SL}_n(\mathbf{R})$ is $\mathrm{SO}_n(\mathbf{R})$. The set of $\mathfrak{so}_n(\mathbf{R})$-invariant scalar products on $\mathbf{R}^n$ is the line generated by the standard scalar product (that it is reduced to a line follows from the absolute irreducibility of the $\mathfrak{so}_n(\mathbf{R})$-action on $\mathbf{R}^n$). Hence this line is $N$-invariant. The stabilizer (in $\mathrm{GL}_n(\mathbf{R})$) of this line is by definition the group of similarities, hence its stabilizer in $\mathrm{SL}_n(\mathbf{R})$ is the group of similarities of determinant 1, that is, $N=\mathrm{SO}_n(\mathbf{R})$. $\Box$

• @Aurel of course, corrected, thanks. (We can erase both comments then) – YCor Jul 1 '16 at 16:14
• I have managed to enlighten myself about Lie groups/algebras enough to understand you proof. However, I still have few questions: 1. In the 3rd from the end paragraph, it should be $\mathfrak{sl_n}$ everywhere, not $\mathfrak{gl_n}$ , right? 2. Could you please give some references for irreducibility of $\mathfrak{p}$? 3. Could you please give some references for $SO_n(R)$ being the stabilizer of $\mathfrak{sl_n}$ in $SL_n(R)$? 4. If I use the fact that I need in my paper, do I put a reference on you, or there is some source in the literature? – erz Jul 19 '16 at 4:50
• I fixed the $gl_n$ typo and added the two requested proofs. This fact is well-known, so it's better if you can find a published reference, but I'm not sure where to find it right now. – YCor Jul 19 '16 at 8:50

First, let us prove that a non-compact subgroup $G$ of $SL_n(\mathbb {R})$ which contains $SO(n)$ is all of $SL_n(\mathbb{R})$. If the connected component $G^0$ is compact, and $SO(n)$ is maximal compact in $SL_n(\mathbb{R})$ , it follows that $G^0=SO(n)$. But it is easy to show that $SO(n)$ it its own normalizer in $SL_n(\mathbb{R})$ and hence $G^0=G$ and is non-compact.

The polar decomposition $SL_n(\mathbb {R})=SO(n)P$ where $P$ is the set of positive definite symmetric matrices of determinant one, shows that $G$ is closed under transposes. We may write the polar decomposition for the Lie algebra $\mathfrak g$ of $G$ as ($\mathfrak{p}_G$ is the space of symmetric matrices in $\mathfrak{g}$): $$\mathfrak{g}=\mathfrak{so}_n\oplus \mathfrak{p}_G.$$ The space $\mathfrak{p}_G$ is non-zero since $G$ is non-compact, and as a representation of $SO(n)$, is a sub-representation of $V=sym^2(std)^0$, where $std$ is the standard representation of $SO(n)$ and where the superscript $^0$ refers to the subspace of trace zero matrices. But, if $n\geq 3$, then $V$ is an irreducible representation of $SO(n)$ and hence $\mathfrak{p}_G=V$. Thus $G=SL_n(\mathbb{R})$. From this Q2 follows easily.

Now consider Q1.Let $G$ be the group generated by your $T$ as well as $O(n)$.By Q2, the image of $G$ in $GL_n(\mathbb{R}/center$ is all of $GL_n(\mathbb{R}/center$. By van der Kallen's observation in the comments, $G\supset SL_n(\mathbb{R})$ and hence for $n\geq 3$, the transitivity statement follows, since $SL_n(\mathbb{R})$ already acts bitransitively.

• Your proof is incomplete, since it assumes implicitly $G$ closed (or, at least, an analytic subgroup). There's some further stuff in between. – YCor Jun 30 '16 at 15:05
• @Ycor; actually, your comments also assume $G$ has a Lie algebra so you have also assumed that $G$ is an analytic subgroup. So your proof is just as incomplete! – Venkataramana Jun 30 '16 at 15:30
• No, I didn't. When I say "the set of $x$ in the Lie algebra...", I mean, in the Lie algebra $\mathfrak{gl}_n$ (but indeed I didn't justify that it's a Lie subalgebra, which requires a little argument). – YCor Jun 30 '16 at 16:06
• I've downvoted your answer because it contains a mistake and you didn't edit it to fix/clarify it. You should say explicitly what you prove, namely that $\mathfrak{so}_n(\mathbf{R)})$ is a maximal subalgebra, and the consequence that $\mathrm{SO}(n)$ is maximal among closed subgroups of $\mathrm{SL}_n(\mathbf{R})$, showing in particular that the closure of $\langle \mathrm{SO}(n),t\rangle$ contains $\mathrm{SL}_n$ whenever $t$ is not a scalar multiple of an isometry. – YCor Jul 3 '16 at 21:40

In fact the subgroup generated by $SO(n)$ and $tSO(n)t^{-1}$ is certainly connected by arc (as the union of images of arc-connected spaces $SO(n)^k, k\in \bf N$ by the obvious product map) So it is an immersed Lie subgroup (Yamabe). But the argument of Venkataramana shows that there is no Lie algebra between $so(n)$ and $sl(n)$, and the Lie algebra of this immersed Lie subgroup is $sl(n)$. By connectedness this group is $SL(n)$.

• This is a good argument. It also shows (thanks, @Thomas!) that there was no "mistake" in the proof that I gave, contrary to the claims of "MO police" . – Venkataramana Jul 20 '16 at 4:59
• @Venkataramana No, instead it fills the gap in your argument :) do you really think that it is police to point out mathematical issues? – YCor Jul 20 '16 at 8:09