Question

Apart from images of representations of subgroups of SU(2), what are the Lie subgroups of SU(3)? Where should I look for a reference?

Answer 1

A first approximation to an answer is to determine the Lie subalgebras of $\mathfrak{su}(3)$. Here is their Hasse diagram with edges denoting inclusions. One can work this out iteratively by working out the maximal subalgebras and this follows from Dynkin's work.

   

The notation $\mathfrak{so}(2)_{[\alpha,\beta]}$ means one element of the pencil of one-dimensional subalgebras of $\mathfrak{so}(2)\oplus \mathfrak{so}(2)$ and $\mathfrak{so}(3)_{\text{irr}}$ is a subalgebra which acts irreducibly on the 3-dimensional irreducible representation.

Answer 2

This is essentially just a slower walk through José's diagram.

I'll assume you're interested in closed, connected subgroups $G\leq SU(3)$. I'll write $V$ for $\mathbb{C}^3$ regarded as a faithful representation of $G$ via the inclusion in $SU(3)$. If $G$ is abelian then we can decompose $V$ as a sum of three one-dimensional representations, say $V=V_1\oplus V_2\oplus V_3$, and then $$ G \leq SU(V)\cap (U(V_1)\times U(V_2)\times U(V_3)) \simeq U(1)\times U(1). $$ All remaining questions about the abelian case are now easy, so I'll assume that $G$ is nonabelian.

Now consider the rank of $G$, ie the dimension of its maximal torus. As $G$ is nonabelian and contained in $SU(3)$ the rank must be one or two.

If the rank is one, then it is standard that $G$ is isomorphic to $SU(2)$ or $SU(2)/\{\pm I\}\simeq SO(3)$. Let $W_1$ denote the standard two-dimensional representation of $SU(2)$, and let $W_2$ denote the symmetric tensor square of $W_1$, which has dimension $3$. The action of $SU(2)$ on $W_2$ factors through $SO(3)$. The only irreducible representations of $SU(2)$ of dimension $\leq 3$ are $\mathbb{C}$, $W_1$ and $W_2$. It follows that one of the following holds:

1. $G\simeq SU(2)$, and $V$ corresponds to $\mathbb{C}\oplus W_1$. This means that there is a one-dimensional subspace $L\leq V$ such that $G=\{g\in SU(V):g|_L=1_L\}$.
2. $G\simeq SO(3)$, and $V$ corresponds to $W_2$. This means that there is a real subspace $X\leq V$ with $V=X\oplus iX$, and $G$ is the evident copy of $SO(X)$ in $SU(V)$.

Suppose instead that the rank is two. This means that any maximal torus in $G$ is also a maximal torus in $SU(V)$, so $G$ is a parabolic subgroup.

Suppose that $V$ is reducible as a representation of $G$. This implies that there is a one-dimensional subspace $L\leq V$ that is preserved by $G$, and thus that $G$ is contained in the image of the homomorphism $\phi:U(L^\perp)\to SU(V)$ given by $$ \phi(g)=g\oplus(\det(g)^{-1}.1_L) $$ As $G$ is connected and nonabelian of rank two, I think it has to be the whole image of $\phi$.

Suppose instead that $V$ is irreducible. I think it then follows from the standard story about parabolic subgroups that $G$ is all of $SU(V)$.