I am not sure about references, and I have only a partial answer to your question. If your group is complex reductive, then it has only finitely many isomorphism classes of irreducible representations if and only if it is itself finite. To see this, we use the Theorem of the Highest Weight to enumerate the irreps by dominant integral weights. There are finitely many of these if and only if the group is finite. For one of these directions, suppose that $G$ is complex reductive and has only finitely many dominant integral weights. Then, its complex dimension must be $0$, so $G$ is discrete. Now, consider a compact real form $K$ of $G$. Note that $K$ is discrete, and hence finite. Since $G$ is the complexification of $K$, $G$ must be finite.
Peter Crooks
- 4.9k
- 2
- 22
- 42