Apart from images of representations of subgroups of SU(2), what are the Lie subgroups of SU(3)? Where should I look for a reference?

1$\begingroup$ Here "Lie subgroup" just amounts to "closed subgroup", which can be connected or not. The closed connected groups can have dimensions up to the total dimension 8, but to enumerate them you should start with some knowledge of standard structure theory as developed in those textbooks on Lie groups which emphasize compact groups. One example is the Springer graduate text by Brocker and tom Dieck, but there are other approaches depending on your background and interests. For instance, J.F. Adams wrote nice lecture notes from a topological viewpoint. $\endgroup$ – Jim Humphreys May 20 '11 at 11:25

$\begingroup$ Why the algebraic topology tag? $\endgroup$ – José FigueroaO'Farrill May 20 '11 at 11:28

1$\begingroup$ Jim: is it clear that Lie subgroup means closed subgroup? There are perfectly nice Lie groups inside SU(3) which are not closed, such as the irrationalslope oneparameter subgroups of the maximal torus. $\endgroup$ – José FigueroaO'Farrill May 20 '11 at 11:34

$\begingroup$ @Jose: Your point is well taken, though the sparse context of the question suggested to me that it was really about closed subgroups. (I'm also unsure what the algebraictopology means here.) $\endgroup$ – Jim Humphreys May 20 '11 at 13:03
A first approximation to an answer is to determine the Lie subalgebras of $\mathfrak{su}(3)$. Here is their Hasse diagram with edges denoting inclusions. One can work this out iteratively by working out the maximal subalgebras and this follows from Dynkin's work.
The notation $\mathfrak{so}(2)_{[\alpha,\beta]}$ means one element of the pencil of onedimensional subalgebras of $\mathfrak{so}(2)\oplus \mathfrak{so}(2)$ and $\mathfrak{so}(3)_{\text{irr}}$ is a subalgebra which acts irreducibly on the 3dimensional irreducible representation.

$\begingroup$ Weirdly, I also tried the
img
tag but for some reason it did not display. I did not have the spaces in front, though. $\endgroup$ – José FigueroaO'Farrill May 20 '11 at 15:06
This is essentially just a slower walk through José's diagram.
I'll assume you're interested in closed, connected subgroups $G\leq SU(3)$. I'll write $V$ for $\mathbb{C}^3$ regarded as a faithful representation of $G$ via the inclusion in $SU(3)$. If $G$ is abelian then we can decompose $V$ as a sum of three onedimensional representations, say $V=V_1\oplus V_2\oplus V_3$, and then $$ G \leq SU(V)\cap (U(V_1)\times U(V_2)\times U(V_3)) \simeq U(1)\times U(1). $$ All remaining questions about the abelian case are now easy, so I'll assume that $G$ is nonabelian.
Now consider the rank of $G$, ie the dimension of its maximal torus. As $G$ is nonabelian and contained in $SU(3)$ the rank must be one or two.
If the rank is one, then it is standard that $G$ is isomorphic to $SU(2)$ or $SU(2)/\{\pm I\}\simeq SO(3)$. Let $W_1$ denote the standard twodimensional representation of $SU(2)$, and let $W_2$ denote the symmetric tensor square of $W_1$, which has dimension $3$. The action of $SU(2)$ on $W_2$ factors through $SO(3)$. The only irreducible representations of $SU(2)$ of dimension $\leq 3$ are $\mathbb{C}$, $W_1$ and $W_2$. It follows that one of the following holds:
 $G\simeq SU(2)$, and $V$ corresponds to $\mathbb{C}\oplus W_1$. This means that there is a onedimensional subspace $L\leq V$ such that $G=\{g\in SU(V):g_L=1_L\}$.
 $G\simeq SO(3)$, and $V$ corresponds to $W_2$. This means that there is a real subspace $X\leq V$ with $V=X\oplus iX$, and $G$ is the evident copy of $SO(X)$ in $SU(V)$.
Suppose instead that the rank is two. This means that any maximal torus in $G$ is also a maximal torus in $SU(V)$, so $G$ is a parabolic subgroup.
Suppose that $V$ is reducible as a representation of $G$. This implies that there is a onedimensional subspace $L\leq V$ that is preserved by $G$, and thus that $G$ is contained in the image of the homomorphism $\phi:U(L^\perp)\to SU(V)$ given by $$ \phi(g)=g\oplus(\det(g)^{1}.1_L) $$ As $G$ is connected and nonabelian of rank two, I think it has to be the whole image of $\phi$.
Suppose instead that $V$ is irreducible. I think it then follows from the standard story about parabolic subgroups that $G$ is all of $SU(V)$.
U(2) is also a subgroup of SU(3). It is missing in the answers above. Probably people mean U(2) when they write SU(2)xU(1). But U(2) is a genuine subgroup of SU(3).

1$\begingroup$ This is better placed as a comment on another user's post. $\endgroup$ – John Pardon Aug 17 '13 at 20:14

$\begingroup$ The Lie algebra of $U(2)$ is mentioned in my answer. I restricted myself to Lie algebras, which is why I said it was a first step in the solution. $\endgroup$ – José FigueroaO'Farrill Dec 1 '15 at 15:24