Apart from images of representations of subgroups of SU(2), what are the Lie subgroups of SU(3)? Where should I look for a reference?

1$\begingroup$ Here "Lie subgroup" just amounts to "closed subgroup", which can be connected or not. The closed connected groups can have dimensions up to the total dimension 8, but to enumerate them you should start with some knowledge of standard structure theory as developed in those textbooks on Lie groups which emphasize compact groups. One example is the Springer graduate text by Brocker and tom Dieck, but there are other approaches depending on your background and interests. For instance, J.F. Adams wrote nice lecture notes from a topological viewpoint. $\endgroup$– Jim HumphreysMay 20, 2011 at 11:25

$\begingroup$ Why the algebraic topology tag? $\endgroup$– José FigueroaO'FarrillMay 20, 2011 at 11:28

3$\begingroup$ Jim: is it clear that Lie subgroup means closed subgroup? There are perfectly nice Lie groups inside SU(3) which are not closed, such as the irrationalslope oneparameter subgroups of the maximal torus. $\endgroup$– José FigueroaO'FarrillMay 20, 2011 at 11:34

$\begingroup$ @Jose: Your point is well taken, though the sparse context of the question suggested to me that it was really about closed subgroups. (I'm also unsure what the algebraictopology means here.) $\endgroup$– Jim HumphreysMay 20, 2011 at 13:03
3 Answers
A first approximation to an answer is to determine the Lie subalgebras of $\mathfrak{su}(3)$. Here is their Hasse diagram with edges denoting inclusions. One can work this out iteratively by working out the maximal subalgebras and this follows from Dynkin's work.
The notation $\mathfrak{so}(2)_{[\alpha,\beta]}$ means one element of the pencil of onedimensional subalgebras of $\mathfrak{so}(2)\oplus \mathfrak{so}(2)$ and $\mathfrak{so}(3)_{\text{irr}}$ is a subalgebra which acts irreducibly on the 3dimensional irreducible representation.

$\begingroup$ For a full list of all maximal Lie subgroups (still not all Lie subgroups but definitely relevant) see math.stackexchange.com/questions/497853/… $\endgroup$ Feb 22 at 20:29

$\begingroup$ Can you expand on the notation with the alpha and beta subscripts? I dont understand what you say following it. $\endgroup$– CraigMay 12 at 3:24

$\begingroup$ These are just the homogeneous coordinates for lines through the origin in the plane. $\endgroup$ May 13 at 7:29
This is essentially just a slower walk through José's diagram.
I'll assume you're interested in closed, connected subgroups $G\leq SU(3)$. I'll write $V$ for $\mathbb{C}^3$ regarded as a faithful representation of $G$ via the inclusion in $SU(3)$. If $G$ is abelian then we can decompose $V$ as a sum of three onedimensional representations, say $V=V_1\oplus V_2\oplus V_3$, and then $$ G \leq SU(V)\cap (U(V_1)\times U(V_2)\times U(V_3)) \simeq U(1)\times U(1). $$ All remaining questions about the abelian case are now easy, so I'll assume that $G$ is nonabelian.
Now consider the rank of $G$, ie the dimension of its maximal torus. As $G$ is nonabelian and contained in $SU(3)$ the rank must be one or two.
If the rank is one, then it is standard that $G$ is isomorphic to $SU(2)$ or $SU(2)/\{\pm I\}\simeq SO(3)$. Let $W_1$ denote the standard twodimensional representation of $SU(2)$, and let $W_2$ denote the symmetric tensor square of $W_1$, which has dimension $3$. The action of $SU(2)$ on $W_2$ factors through $SO(3)$. The only irreducible representations of $SU(2)$ of dimension $\leq 3$ are $\mathbb{C}$, $W_1$ and $W_2$. It follows that one of the following holds:
 $G\simeq SU(2)$, and $V$ corresponds to $\mathbb{C}\oplus W_1$. This means that there is a onedimensional subspace $L\leq V$ such that $G=\{g\in SU(V):g_L=1_L\}$.
 $G\simeq SO(3)$, and $V$ corresponds to $W_2$. This means that there is a real subspace $X\leq V$ with $V=X\oplus iX$, and $G$ is the evident copy of $SO(X)$ in $SU(V)$.
Suppose instead that the rank is two. This means that any maximal torus in $G$ is also a maximal torus in $SU(V)$, so $G$ is a parabolic subgroup.
Suppose that $V$ is reducible as a representation of $G$. This implies that there is a onedimensional subspace $L\leq V$ that is preserved by $G$, and thus that $G$ is contained in the image of the homomorphism $\phi:U(L^\perp)\to SU(V)$ given by $$ \phi(g)=g\oplus(\det(g)^{1}.1_L) $$ As $G$ is connected and nonabelian of rank two, I think it has to be the whole image of $\phi$.
Suppose instead that $V$ is irreducible. I think it then follows from the standard story about parabolic subgroups that $G$ is all of $SU(V)$.
U(2) is also a subgroup of SU(3). It is missing in the answers above. Probably people mean U(2) when they write SU(2)xU(1). But U(2) is a genuine subgroup of SU(3).

3$\begingroup$ This is better placed as a comment on another user's post. $\endgroup$ Aug 17, 2013 at 20:14

4$\begingroup$ The Lie algebra of $U(2)$ is mentioned in my answer. I restricted myself to Lie algebras, which is why I said it was a first step in the solution. $\endgroup$ Dec 1, 2015 at 15:24