Question

Given the group SU(N) of NxN unitary matrices, does there exist a subgroup S with a manifold dimension larger than the SU(N-1) manifold dimension and smaller than the SU(N) one? S should not necessarily have SU(N-1) as subgroup.

Answer 1

Your question is equivalent to the question about maximal dimension of a proper Lie subalgebra of $su(N)$. Clearly such subalgebra is reductive since it has a positive definite invariant form. Thus you are looking for a reductive Lie algebra of maximal possible dimension strictly smaller than $N^2-1$ with faithful representation $V$ of dimension $N$. We claim that this dimension is $(N-1)^2$ for $N>4$. Here is a sketch of argument:

1) representation $V$ is irreducible: if it splits into direct sum of subrepresentations of dimensions $a$ and $b$, then dimension of our Lie algebra is less than $a^2+b^2\le (N-1)^2$ (one needs to consider the special case when $a=1$ separately).

2) representation $V$ is not a tensor product of irreducible representations: if it is a tensor product of representations of dimensions $a$ and $b$, then our Lie algebra has dimension less than $a^2+b^2\le (N/2)^2+(N/2)^2=N^2/2\le (N-1)^2$.

3) it follows from 2) above that our Lie algebra is in fact simple Lie algebra. The smallest possible dimension of an irreducible representation of any simple Lie algebra is well known (this is always one of the fundamental representations). Now quick search gives you the result (and the counterexample pointed out by Somnath).

Answer 2

I guess the answer is yes. Consider the symplectic group $Sp(2)$, consiting of elements in $GL_2(\mathbb{H})$ which preserve the Hermitian form on $\mathbb{H}^2$. It is a subgroup of $SU(4)$ and has dimension $10$. However, there is no inclusion of groups $\iota:SU(3)\to Sp(2)$ since the quotient $X:=Sp(2)/SU(3)$ is a homogeneous space and a closed manifold of dimension $2$ with trivial $\pi_1$ and $\pi_2$.

Answer 3

The answer is yes for all $n$, $SU(n)$ contains $U(n-1)$ as a block-diagonal $(n-1)\times (n-1)+1\times 1$ matrix. This $U(n-1)$ contains $SU(n-1)$ of cours.

For $n$ large enough it should be possible to prove that there are no more examples using classification of compact groups.