Hi! I would like to know if there is an explicit classification of the algebraic (i.e., Zariski closed) subgroups of the symplectic group Sp(4,R) and/or more generally Sp(2n,R) somewhere in the literature.

1$\begingroup$ Would you care to explain why you want this? I'm sure Sp(4,R) can be done by hand (though getting all the finite groups might be hard; just considering connected groups will make your life much easier), but I'm just having a hard time imagining where knowing this will get you. $\endgroup$– Ben Webster ♦Commented Dec 19, 2009 at 18:08

3$\begingroup$ Of course I can (try to) explain it I'm interested in the positivity of Lyapunov exponents of linear cocycles over chaotic dynamical systems, more precisely, my coauthors and I are looking at the KontsevichZorich (KZ) cocycle over the Teichmuller flow on the moduli space of Abelian differentials on Riemann surfaces. In genus 3, it turns out that this cocycle is a Sp(4,R) cocycle. After the works of Goldsheid and Margulis, it suffices to show that the monoid of matrices related to KZ cocycle is Zariski dense in Sp(4,R) and during this task it could be useful to know the answer to my question. $\endgroup$– MatheusCommented Dec 19, 2009 at 23:31

$\begingroup$ Is R here the reals or an arbitrary ring? $\endgroup$– Daniel SebaldCommented Apr 21 at 20:29
3 Answers
On the one hand, I could not find a published answer with a cursory search. On the other hand, as Ben says, you could work out the answer "by hand". Instead of writing down a sheer list, which might be complicated (and I haven't done the work), I'll write down the main ingredients.
A Zariskiclosed subgroup $H$ of any connected semisimple Lie group $G$ has three pieces: (1) finite, (2) connected semisimple, and (3) connected solvable. The Zariski topology forces $H$ to have only finitely many components; if $H_0$ is the connected subgroup, then $H/H_0$ is the finite piece. Then the Lie algebra of $H_0$ has a Levi decomposition, so that you get the other two pieces. The way to analyze the question is to chase down the possibilities for all three pieces.
I think that the finite part always lifts to a slightly larger finite subgroup of $H$. This is not true for groups in general, but I think that it is true in context. Then this finite group is contained in a maximal compact group of $G$. Happily, the compact core of $\text{Sp}(4,\mathbb{R})$ is $\text{SU}(2)$, and the finite subgroups are classified by simply laced Dynkin diagrams.
A semisimple, connected subgroup of $G$ corresponds to a semisimple Lie subalgebra, and that complexifies. The Lie algebra $\text{sp}(4,\mathbb{C})$ does not have very many inequivalent semisimple subalgebras. From looking a rank, they are isomorphic to $\text{sl}(2,\mathbb{C})$ or $\text{sl}(2,\mathbb{C}) \oplus \text{sl}(2,\mathbb{C})$. I am confusing myself a little with the possible positions of the former, although I know there are only a few. The latter embeds in only one way. Then you would work backwards to get the real forms of these complex subalgebras; again there wouldn't be very many.
Finally the solvable part also complexifies and I think that it is contained in a Borel subalgebra at the Lie algebra level.
As for the more general question, for $\text{Sp}(2n,\mathbb{R})$, there is a tidy converse answer that also shows you that you can't expect a tidy answer for all fixed $n$. Namely, if $G$ is any algebraic group, you can classify its antiselfdual (or symplectically selfdual) representations. Every algebraic group will have some, because every algebraic group has representations in $\text{GL}(n,\mathbb{R})$. A more interesting case is when $G$ has an irreducible symplectically selfdual representation. For that purpose, you check that the irreducible representation is real, and then check the Frobenius–Schur indicator.

$\begingroup$ Dear Greg, thanks! As I suspected, this can't be easily located in the literature. Indeed, I posted this question since beside the examples of SL_2xSL_2 and tensor product of SL_2 and SO_2, Don Zagier showed me a funny way to put the 3rd symmetric power of SL_2 inside Sp(4). By the way, let me mention that the finite subgroups appearing in the specific case of KZ cocycle on Teich curves is known: after the work of Moller, KZ cocycle can act with finite groups only in genus 3, 4 and 5, and in the genus 3 and 4, it acts over two special examples with the Weyl group of a root system of D_4 type. $\endgroup$– MatheusCommented Dec 20, 2009 at 10:43

1$\begingroup$ What you must mean is that you can embed SL(2) in Sp(4) using the third symmetric power of the defining representation of SL(2). It's not the third symmetric power of SL(2) itself. Yes, I thought of that embedding. I only wasn't sure whether I had everything semisimple, and what was different from what. $\endgroup$ Commented Dec 20, 2009 at 16:55

$\begingroup$ Ops, you're absolutely right... By the way, do you mind of sharing your list of semisimple here? $\endgroup$– MatheusCommented Dec 20, 2009 at 18:17

$\begingroup$ You're probably not too interested in the finite subgroups of $Sp_{4}(\mathbb{C})$ I guess; but as mentioned above with the simply laced Dynkin diagrams, a good reference is Miles Reid's notes on the topic: arxiv.org/abs/alggeom/9702016. $\endgroup$ Commented Dec 21, 2009 at 5:10
The paper `On the subgroup description of classical groups' by Martin Liebeck and Gary Seitz (available at http://dx.doi.org/10.1007/s002220050270) gives a structure description of the closed subgroups of classical groups over an algebraically closed field. The subgroups are either the stabilisers of a subspace, subspace decomposition or tensor product decomposition, or a classical group, or modulo scalars is the normaliser of an elementary abelian $r$group, or modulo scalars is almost simple. It generalises the result by Aschbacher in the finite field case.
Maximal (connected, I think) subgroups of complex classical groups have been classified by E.B. Dynkin in the early 50's. Here is a link to the MR of the russian paper, translated in Amer. Math. Soc. Transl., Series 2, Vol. 6 (1957), 245378 (which isn't in MR). Then T.M. Selim (found in the citations of Dynkin's paper) undertook the real case (see here), but the summary in MR has strange notations and what is exactly proved is not clear to me. The paper by Liebeck and Seitz cited in Michael's answer has MR here, but the link to the article in MR (given in Michael's answer) seems broken. By the way, it is in Inventiones 134 (1998), no. 2, 427453.

$\begingroup$ I just discovered that E.B. Dynkin has an arXiv paper with A. Minchenko arxiv.org/abs/0909.2289 where they improve the methods of another Dynkin 1952 paper on semisimple subalgebras, and makes some corrections. It covers "only" the ADE cases, the remaining ones "will be the subject of the next publication". $\endgroup$– BS.Commented Jul 6, 2010 at 14:22