A question on linear groups

Question

Asking this question I have made a mistake joining my main question with two simple ones, so it hasn't received enough attention, however there was a partial answer, which was not elaborated, and now I am completely lost =)

My original question goes as follows (with a little editing).

Q1: Suppose we have a linear operator $T:\mathbb{R}^n\to \mathbb{R}^n$, $n>2$, which is NOT a constant times an isometry. Let $G$ be the subgroup of $GL_n(\mathbb{R})$, generated by the orthogonal group together with $T$. I need to prove (or refute) that $G$ acts "bitransitively" on $\mathbb{R}^n$, in the sense that it can map any pair of non-colinear vectors into any other pair of non-colinear vectors. I also need a similar result for the complex case.

A comment that I received states that this is true, due the following:

Q2: $\mathbb{R}^*O_n(\mathbb{R})$ is a maximal subgroup in $GL_n(\mathbb{R})$ (and analogously for the complex case).

I don't really see neither why these statements hold, nor how to derive my question from them, except of the case when $\det T=\pm1$. I have found two papers that seem to have proofs of these statements, but I don't really have a background in algebra, so I can't even understand if this is the case (I am not sure if the Euclidean spaces fit into the scope of these papers). Furthermore, I am convinced that there should be some geometry behind this issue, and there is no chance that I can retrieve the geometric intuition from theses papers, again since my background doesn't allow me to read them. And as I've mentioned before, I don't see how to derive Q1 from Q2 in general case.

I hope you can help me with this predicament.

I will also add another related (vaguely stated) question.

Q3: What can be said about a subgroup $G$ of $GL_n(\mathbb{R})$, if it can map any $k$-tuple of linearly independent vectors into any other $k$-tuple, for some $k\in\overline{1,n-1}$?

Answer 1

Here's a complete proof of:

Every subgroup of $\mathrm{GL}_n(\mathbf{R})$ containing $\mathrm{SO}_n(\mathbf{R})$ is either contained in the group of similarities $\mathbf{R}^*\mathrm{O}_n(\mathbf{R})$, or contains $\mathrm{SL}_n(\mathbf{R})$.

(This is equivalent to the statement that for every $t\in\mathrm{GL}_n(\mathbf{R})\smallsetminus \mathbf{R}^*\mathrm{O}(n)$, the subgroup generated by $\mathrm{SO}_n(\mathbf{R})\cup\{t\}$ contains $\mathrm{SL}_n(\mathbf{R})$. The answers to Q2 for $n\ge 2$ and to Q1 for $n\ge 3$ immediately follow.)

Let $H$ be a subgroup of $\mathrm{GL}_n(\mathbf{R})$. Let $\mathfrak{h}$ be the set of $x\in\mathfrak{gl}_n(\mathbf{R})$ such that there exists a $C^1$ map $u:\mathbf{R}\to\mathrm{GL}_n(\mathbf{R})$ such that $u(0)=I_n$ (the identity matrix), $u'(0)=x$, and $u$ takes values in $H$. Then $\mathfrak{h}$ is a linear subspace. Indeed, it's clearly stable under scalar multiplication, and if $u,v$ are two such maps with $u'(0)=x$, $v'(0)=y$, then defining $w(t)=u(t)v(t)$ we have $w'(0)=x+y$.

Now assume that $H$ contains $\mathrm{SO}_n(\mathbf{R})$.

First assume that $H$ is contained in $\mathrm{SL}_n(\mathbf{R})$; it follows that $\mathfrak{h}$ is contained in $\mathfrak{sl}_n(\mathbf{R})$.

Then $\mathfrak{h}$ is invariant under the action by conjugation of $\mathrm{SO}_n(\mathbf{R})$. Now (as mentioned in the post by Venkataramana), we can decompose $\mathfrak{sl}_n(\mathbf{R})$ as a $\mathrm{SO}_n(\mathbf{R})$-representation, namely $\mathfrak{sl}_n(\mathbf{R}) = \mathfrak{so}_n(\mathbf{R})\oplus\mathfrak{p}$, where $\mathfrak{p}$ is the set of symmetric matrices with trace zero. Then for all $n \ge 2$ (including $n=2$), $\mathfrak{p}$ is irreducible (see a proof below); it follows that the subspaces of $\mathfrak{sl}_n(\mathbf{R})$ containing $\mathfrak{so}_n(\mathbf{R})$ and invariant under conjugation by $\mathrm{SO}_n(\mathbf{R})$ are $\mathfrak{so}_n(\mathbf{R})$ and $\mathfrak{so}_n(\mathbf{R})\oplus\mathfrak{p}=\mathfrak{sl}_n(\mathbf{R})$.

If $H$ is contained in $\mathrm{SO}_n(\mathbf{R})$ then we are in one of the first cases. Conversely, if $H$ is not contained in $\mathrm{SO}_n(\mathbf{R})$ then since the latter is the stabilizer of $\mathfrak{so}_n(\mathbf{R})$ in $\mathrm{SL}_n(\mathbf{R})$ (see a proof below), we deduce that $\mathfrak{h}$ contains a conjugate of $\mathfrak{so}_n(\mathbf{R})$ distinct from $\mathfrak{so}_n$. Hence we are in the second case, that is, $\mathfrak{h}=\mathfrak{sl}_n(\mathbf{R})$. In this case we can pick a basis $(e_1,\dots,e_m)$ of $\mathfrak{sl}_n(\mathbf{R})$ ($m=n^2-1$) and functions $u_1,\dots,u_m:\mathbf{R}\to H$ as in the definition of $\mathfrak{h}$ with $u'_i(0)=e_i$. It follow that the differential of the function $(u_1,\dots,u_m):\mathbf{R}^m\to H\subset\mathrm{SL}_n(\mathbf{R})$ at zero is surjective. Hence its image contains a neighborhood of $I_n$ in $\mathrm{SL}_n(\mathbf{R})$. Thus $H$ is open in $\mathrm{SL}_n(\mathbf{R})$ and hence by connectedness of the latter, is equal to $\mathrm{SL}_n(\mathbf{R})$.

Now remove the assumption that $H$ is in $\mathrm{SL}_n(\mathbf{R})$. If $H\cap\mathrm{SL}_n(\mathbf{R})=\mathrm{SO}_n(\mathbf{R})$, then $H$ is contained in the normalizer $\mathbf{R}^*\mathrm{O}_n(\mathbf{R})$ of $\mathrm{SO}_n(\mathbf{R})$. Otherwise, we deduce from the previous case that $H\cap\mathrm{SL}_n(\mathbf{R})$ equals $\mathrm{SL}_n(\mathbf{R})$, that is to say, $H$ contains $\mathrm{SL}_n(\mathbf{R})$. This finishes the proof.

---

Addendum: proof of some basic facts (asked in a comment)

Proof that the action of $\mathrm{SO}_n(\mathbf{R})$ on $\mathfrak{p}$ is irreducible for $n\ge 2$. First, consider the action of the subgroup of signed permutation matrices with determinant 1 on the subspace $D\subset\mathfrak{p}$ of diagonal matrices with trace zero. It factors through the action of permutation matrices (not only even ones, using $n\ge 2$) on the set of $n$-vectors with sum zero (by permutation of coordinates), and this action is well-known and easily checked to be irreducible.

Now let $V$ be a nonzero $\mathrm{SO}_n(\mathbf{R})$-invariant subspace of $\mathfrak{p}$. Picking a nonzero element in $V$, it has a $\mathrm{SO}_n(\mathbf{R})$-conjugate that is diagonal. Therefore $V\cap D\neq 0$, and by the previous irreducibility fact, $D\subset V$. Now any element of $\mathfrak{p}$ has a $\mathrm{SO}_n(\mathbf{R})$-conjugate that is diagonal, so $V=\mathfrak{p}$. $\Box$

Proof that the stabilizer $N$ of $\mathfrak{so}_n(\mathbf{R})$ in $\mathrm{SL}_n(\mathbf{R})$ is $\mathrm{SO}_n(\mathbf{R})$. The set of $\mathfrak{so}_n(\mathbf{R})$-invariant scalar products on $\mathbf{R}^n$ is the line generated by the standard scalar product (that it is reduced to a line follows from the absolute irreducibility of the $\mathfrak{so}_n(\mathbf{R})$-action on $\mathbf{R}^n$). Hence this line is $N$-invariant. The stabilizer (in $\mathrm{GL}_n(\mathbf{R})$) of this line is by definition the group of similarities, hence its stabilizer in $\mathrm{SL}_n(\mathbf{R})$ is the group of similarities of determinant 1, that is, $N=\mathrm{SO}_n(\mathbf{R})$. $\Box$

Answer 2

First, let us prove that a non-compact subgroup $G$ of $SL_n(\mathbb {R})$ which contains $SO(n)$ is all of $SL_n(\mathbb{R})$. If the connected component $G^0$ is compact, and $SO(n)$ is maximal compact in $SL_n(\mathbb{R})$ , it follows that $G^0=SO(n)$. But it is easy to show that $SO(n)$ it its own normalizer in $SL_n(\mathbb{R})$ and hence $G^0=G$ and is non-compact.

The polar decomposition $SL_n(\mathbb {R})=SO(n)P$ where $P$ is the set of positive definite symmetric matrices of determinant one, shows that $G$ is closed under transposes. We may write the polar decomposition for the Lie algebra $\mathfrak g$ of $G$ as ($\mathfrak{p}_G$ is the space of symmetric matrices in $\mathfrak{g}$): $$\mathfrak{g}=\mathfrak{so}_n\oplus \mathfrak{p}_G.$$ The space $\mathfrak{p}_G$ is non-zero since $G$ is non-compact, and as a representation of $SO(n)$, is a sub-representation of $V=sym^2(std)^0$, where $std$ is the standard representation of $SO(n)$ and where the superscript $^0$ refers to the subspace of trace zero matrices. But, if $n\geq 3$, then $V$ is an irreducible representation of $SO(n)$ and hence $\mathfrak{p}_G=V$. Thus $G=SL_n(\mathbb{R})$. From this Q2 follows easily.

Now consider Q1.Let $G$ be the group generated by your $T$ as well as $O(n)$.By Q2, the image of $G$ in $GL_n(\mathbb{R}/center$ is all of $GL_n(\mathbb{R}/center$. By van der Kallen's observation in the comments, $G\supset SL_n(\mathbb{R})$ and hence for $n\geq 3$, the transitivity statement follows, since $SL_n(\mathbb{R})$ already acts bitransitively.

Answer 3

In fact the subgroup generated by $SO(n)$ and $tSO(n)t^{-1}$ is certainly connected by arc (as the union of images of arc-connected spaces $SO(n)^k, k\in \bf N$ by the obvious product map) So it is an immersed Lie subgroup (Yamabe). But the argument of Venkataramana shows that there is no Lie algebra between $so(n)$ and $sl(n)$, and the Lie algebra of this immersed Lie subgroup is $sl(n)$. By connectedness this group is $SL(n)$.