Asking this question I have made a mistake joining my main question with two simple ones, so it hasn't received enough attention, however there was a partial answer, which was not elaborated, and now I am completely lost =)

My original question goes as follows (with a little editing).

Q1: Suppose we have a linear operator $T:\mathbb{R}^n\to \mathbb{R}^n$, $n>2$, which is NOT a constant times an isometry. Let $G$ be the subgroup of $GL_n(\mathbb{R})$, generated by the orthogonal group together with $T$. I need to prove (or refute) that $G$ acts "bitransitively" on $\mathbb{R}^n$, in the sense that it can map any pair of non-colinear vectors into any other pair of non-colinear vectors. I also need a similar result for the complex case.

A comment that I received states that this is true, due the following:

Q2: $\mathbb{R}^*O_n(\mathbb{R})$ is a maximal subgroup in $GL_n(\mathbb{R})$ (and analogously for the complex case).

I don't really see neither why these statements hold, nor how to derive my question from them, except of the case when $\det T=\pm1$. I have found two papers that seem to have proofs of these statements, but I don't really have a background in algebra, so I can't even understand if this is the case (I am not sure if the Euclidean spaces fit into the scope of these papers). Furthermore, I am convinced that there should be some geometry behind this issue, and there is no chance that I can retrieve the geometric intuition from theses papers, again since my background doesn't allow me to read them. And as I've mentioned before, I don't see how to derive Q1 from Q2 in general case.

I hope you can help me with this predicament.

I will also add another related (vaguely stated) question.

Q3: What can be said about a subgroup $G$ of $GL_n(\mathbb{R})$, if it can map any $k$-tuple of linearly independent vectors into any other $k$-tuple, for some $k\in\overline{1,n-1}$?

  • $\begingroup$ For $n=2$ it is wrong. Take $T$ of determinant one. $\endgroup$ – Wilberd van der Kallen Jun 30 '16 at 7:25
  • $\begingroup$ Q2 holds true: indeed here determinant does not matter since the subgroup already contains all homotheties and contains elements of all determinants. After modding out by homotheties, the question boils down to whether $PO(n)(R)$ is maximal in $PGLn(R)$, and this is true and follows from the fact that $SO(n)(R)$ is maximal in $SL_n(R)$. Here $R$ are the reals. $\endgroup$ – YCor Jun 30 '16 at 7:50
  • $\begingroup$ @WilberdvanderKallen I first misread your comment and I refer to Q2 $\endgroup$ – YCor Jun 30 '16 at 7:50
  • $\begingroup$ I should have said that Q1 is wrong for $n=2$. $\endgroup$ – Wilberd van der Kallen Jun 30 '16 at 7:54
  • $\begingroup$ Indeed. Still for all $n$ it's true that a subgroup containing the orthogonal group and not contained in the group of similarities, will contain $SL_n(R)$. Thus Q1 is true for $n\ge 3$. $\endgroup$ – YCor Jun 30 '16 at 8:12

Here's a complete proof of:

Every subgroup of $\mathrm{GL}_n(\mathbf{R})$ containing $\mathrm{SO}_n(\mathbf{R})$ is either contained in the group of similarities $\mathbf{R}^*\mathrm{O}_n(\mathbf{R})$, or contains $\mathrm{SL}_n(\mathbf{R})$.

(This is equivalent to the statement that for every $t\in\mathrm{GL}_n(\mathbf{R})\smallsetminus \mathbf{R}^*\mathrm{O}(n)$, the subgroup generated by $\mathrm{SO}_n(\mathbf{R})\cup\{t\}$ contains $\mathrm{SL}_n(\mathbf{R})$. The answers to Q2 for $n\ge 2$ and to Q1 for $n\ge 3$ immediately follow.)

Let $H$ be a subgroup of $\mathrm{GL}_n(\mathbf{R})$. Let $\mathfrak{h}$ be the set of $x\in\mathfrak{gl}_n(\mathbf{R})$ such that there exists a $C^1$ map $u:\mathbf{R}\to\mathrm{GL}_n(\mathbf{R})$ such that $u(0)=I_n$ (the identity matrix), $u'(0)=x$, and $u$ takes values in $H$. Then $\mathfrak{h}$ is a linear subspace. Indeed, it's clearly stable under scalar multiplication, and if $u,v$ are two such maps with $u'(0)=x$, $v'(0)=y$, then defining $w(t)=u(t)v(t)$ we have $w'(0)=x+y$.

Now assume that $H$ contains $\mathrm{SO}_n(\mathbf{R})$.

First assume that $H$ is contained in $\mathrm{SL}_n(\mathbf{R})$; it follows that $\mathfrak{h}$ is contained in $\mathfrak{sl}_n(\mathbf{R})$.

Then $\mathfrak{h}$ is invariant under the action by conjugation of $\mathrm{SO}_n(\mathbf{R})$. Now (as mentioned in the post by Venkataramana), we can decompose $\mathfrak{sl}_n(\mathbf{R})$ as a $\mathrm{SO}_n(\mathbf{R})$-representation, namely $\mathfrak{sl}_n(\mathbf{R}) = \mathfrak{so}_n(\mathbf{R})\oplus\mathfrak{p}$, where $\mathfrak{p}$ is the set of symmetric matrices with trace zero. Then for all $n \ge 2$ (including $n=2$), $\mathfrak{p}$ is irreducible (see a proof below); it follows that the subspaces of $\mathfrak{sl}_n(\mathbf{R})$ containing $\mathfrak{so}_n(\mathbf{R})$ and invariant under conjugation by $\mathrm{SO}_n(\mathbf{R})$ are $\mathfrak{so}_n(\mathbf{R})$ and $\mathfrak{so}_n(\mathbf{R})\oplus\mathfrak{p}=\mathfrak{sl}_n(\mathbf{R})$.

If $H$ is contained in $\mathrm{SO}_n(\mathbf{R})$ then we are in one of the first cases. Conversely, if $H$ is not contained in $\mathrm{SO}_n(\mathbf{R})$ then since the latter is the stabilizer of $\mathfrak{so}_n(\mathbf{R})$ in $\mathrm{SL}_n(\mathbf{R})$ (see a proof below), we deduce that $\mathfrak{h}$ contains a conjugate of $\mathfrak{so}_n(\mathbf{R})$ distinct from $\mathfrak{so}_n$. Hence we are in the second case, that is, $\mathfrak{h}=\mathfrak{sl}_n(\mathbf{R})$. In this case we can pick a basis $(e_1,\dots,e_m)$ of $\mathfrak{sl}_n(\mathbf{R})$ ($m=n^2-1$) and functions $u_1,\dots,u_m:\mathbf{R}\to H$ as in the definition of $\mathfrak{h}$ with $u'_i(0)=e_i$. It follow that the differential of the function $(u_1,\dots,u_m):\mathbf{R}^m\to H\subset\mathrm{SL}_n(\mathbf{R})$ at zero is surjective. Hence its image contains a neighborhood of $I_n$ in $\mathrm{SL}_n(\mathbf{R})$. Thus $H$ is open in $\mathrm{SL}_n(\mathbf{R})$ and hence by connectedness of the latter, is equal to $\mathrm{SL}_n(\mathbf{R})$.

Now remove the assumption that $H$ is in $\mathrm{SL}_n(\mathbf{R})$. If $H\cap\mathrm{SL}_n(\mathbf{R})=\mathrm{SO}_n(\mathbf{R})$, then $H$ is contained in the normalizer $\mathbf{R}^*\mathrm{O}_n(\mathbf{R})$ of $\mathrm{SO}_n(\mathbf{R})$. Otherwise, we deduce from the previous case that $H\cap\mathrm{SL}_n(\mathbf{R})$ equals $\mathrm{SL}_n(\mathbf{R})$, that is to say, $H$ contains $\mathrm{SL}_n(\mathbf{R})$. This finishes the proof.

Addendum: proof of some basic facts (asked in a comment)

Proof that the action of $\mathrm{SO}_n(\mathbf{R})$ on $\mathfrak{p}$ is irreducible for $n\ge 2$. First, consider the action of the subgroup of signed permutation matrices with determinant 1 on the subspace $D\subset\mathfrak{p}$ of diagonal matrices with trace zero. It factors through the action of permutation matrices (not only even ones, using $n\ge 2$) on the set of $n$-vectors with sum zero (by permutation of coordinates), and this action is well-known and easily checked to be irreducible.

Now let $V$ be a nonzero $\mathrm{SO}_n(\mathbf{R})$-invariant subspace of $\mathfrak{p}$. Picking a nonzero element in $V$, it has a $\mathrm{SO}_n(\mathbf{R})$-conjugate that is diagonal. Therefore $V\cap D\neq 0$, and by the previous irreducibility fact, $D\subset V$. Now any element of $\mathfrak{p}$ has a $\mathrm{SO}_n(\mathbf{R})$-conjugate that is diagonal, so $V=\mathfrak{p}$. $\Box$

Proof that the stabilizer $N$ of $\mathfrak{so}_n(\mathbf{R})$ in $\mathrm{SL}_n(\mathbf{R})$ is $\mathrm{SO}_n(\mathbf{R})$. The set of $\mathfrak{so}_n(\mathbf{R})$-invariant scalar products on $\mathbf{R}^n$ is the line generated by the standard scalar product (that it is reduced to a line follows from the absolute irreducibility of the $\mathfrak{so}_n(\mathbf{R})$-action on $\mathbf{R}^n$). Hence this line is $N$-invariant. The stabilizer (in $\mathrm{GL}_n(\mathbf{R})$) of this line is by definition the group of similarities, hence its stabilizer in $\mathrm{SL}_n(\mathbf{R})$ is the group of similarities of determinant 1, that is, $N=\mathrm{SO}_n(\mathbf{R})$. $\Box$

| cite | improve this answer | |
  • $\begingroup$ @Aurel of course, corrected, thanks. (We can erase both comments then) $\endgroup$ – YCor Jul 1 '16 at 16:14
  • $\begingroup$ I have managed to enlighten myself about Lie groups/algebras enough to understand you proof. However, I still have few questions: 1. In the 3rd from the end paragraph, it should be $\mathfrak{sl_n}$ everywhere, not $\mathfrak{gl_n}$ , right? 2. Could you please give some references for irreducibility of $\mathfrak{p}$? 3. Could you please give some references for $SO_n(R)$ being the stabilizer of $\mathfrak{sl_n}$ in $SL_n(R)$? 4. If I use the fact that I need in my paper, do I put a reference on you, or there is some source in the literature? $\endgroup$ – erz Jul 19 '16 at 4:50
  • $\begingroup$ I fixed the $gl_n$ typo and added the two requested proofs. This fact is well-known, so it's better if you can find a published reference, but I'm not sure where to find it right now. $\endgroup$ – YCor Jul 19 '16 at 8:50

First, let us prove that a non-compact subgroup $G$ of $SL_n(\mathbb {R})$ which contains $SO(n)$ is all of $SL_n(\mathbb{R})$. If the connected component $G^0$ is compact, and $SO(n)$ is maximal compact in $SL_n(\mathbb{R})$ , it follows that $G^0=SO(n)$. But it is easy to show that $SO(n)$ it its own normalizer in $SL_n(\mathbb{R})$ and hence $G^0=G$ and is non-compact.

The polar decomposition $SL_n(\mathbb {R})=SO(n)P$ where $P$ is the set of positive definite symmetric matrices of determinant one, shows that $G$ is closed under transposes. We may write the polar decomposition for the Lie algebra $\mathfrak g$ of $G$ as ($\mathfrak{p}_G$ is the space of symmetric matrices in $\mathfrak{g}$): $$\mathfrak{g}=\mathfrak{so}_n\oplus \mathfrak{p}_G.$$ The space $\mathfrak{p}_G$ is non-zero since $G$ is non-compact, and as a representation of $SO(n)$, is a sub-representation of $V=sym^2(std)^0$, where $std$ is the standard representation of $SO(n)$ and where the superscript $^0$ refers to the subspace of trace zero matrices. But, if $n\geq 3$, then $V$ is an irreducible representation of $SO(n)$ and hence $\mathfrak{p}_G=V$. Thus $G=SL_n(\mathbb{R})$. From this Q2 follows easily.

Now consider Q1.Let $G$ be the group generated by your $T$ as well as $O(n)$.By Q2, the image of $G$ in $GL_n(\mathbb{R}/center$ is all of $GL_n(\mathbb{R}/center$. By van der Kallen's observation in the comments, $G\supset SL_n(\mathbb{R})$ and hence for $n\geq 3$, the transitivity statement follows, since $SL_n(\mathbb{R})$ already acts bitransitively.

| cite | improve this answer | |
  • $\begingroup$ Your proof is incomplete, since it assumes implicitly $G$ closed (or, at least, an analytic subgroup). There's some further stuff in between. $\endgroup$ – YCor Jun 30 '16 at 15:05
  • $\begingroup$ @Ycor; actually, your comments also assume $G$ has a Lie algebra so you have also assumed that $G$ is an analytic subgroup. So your proof is just as incomplete! $\endgroup$ – Venkataramana Jun 30 '16 at 15:30
  • $\begingroup$ No, I didn't. When I say "the set of $x$ in the Lie algebra...", I mean, in the Lie algebra $\mathfrak{gl}_n$ (but indeed I didn't justify that it's a Lie subalgebra, which requires a little argument). $\endgroup$ – YCor Jun 30 '16 at 16:06
  • $\begingroup$ I've downvoted your answer because it contains a mistake and you didn't edit it to fix/clarify it. You should say explicitly what you prove, namely that $\mathfrak{so}_n(\mathbf{R)})$ is a maximal subalgebra, and the consequence that $\mathrm{SO}(n)$ is maximal among closed subgroups of $\mathrm{SL}_n(\mathbf{R})$, showing in particular that the closure of $\langle \mathrm{SO}(n),t\rangle$ contains $\mathrm{SL}_n$ whenever $t$ is not a scalar multiple of an isometry. $\endgroup$ – YCor Jul 3 '16 at 21:40

In fact the subgroup generated by $SO(n)$ and $tSO(n)t^{-1}$ is certainly connected by arc (as the union of images of arc-connected spaces $SO(n)^k, k\in \bf N$ by the obvious product map) So it is an immersed Lie subgroup (Yamabe). But the argument of Venkataramana shows that there is no Lie algebra between $so(n)$ and $sl(n)$, and the Lie algebra of this immersed Lie subgroup is $sl(n)$. By connectedness this group is $SL(n)$.

| cite | improve this answer | |
  • $\begingroup$ This is a good argument. It also shows (thanks, @Thomas!) that there was no "mistake" in the proof that I gave, contrary to the claims of "MO police" . $\endgroup$ – Venkataramana Jul 20 '16 at 4:59
  • $\begingroup$ @Venkataramana No, instead it fills the gap in your argument :) do you really think that it is police to point out mathematical issues? $\endgroup$ – YCor Jul 20 '16 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.