$\newcommand{\G}{\mathcal{G}}
\newcommand{\K}{\mathcal{K}} $Question: When does $ \G $ admit a finite maximal closed subgroup?
Answer : Must be one of the following two cases
- $ \G $ is compact and simple
- $ \G $ is not compact in which case $ \G $ cannot be connected and moreover the component group $ \G/\G^\circ $ does not preserve any nontrivial proper closed subgroup (see comment from YCor about $ C_5 \ltimes \mathbb{R}^2 $).
From now on I will confine myself to the case that $ \G $ is connected.
In other words I will consider the statement "A connected Lie group $ \G $ has a finite maximal closed subgroup $ G $ if and only if $ \G $ is compact and simple."
The first implication is true.
Claim 1:
If a connected Lie group $ \G $ has a finite maximal closed subgroup $ G $ then $ \G $ must be compact and simple.
Proof:
Let $ \G $ be a connected Lie group and $ G $ a finite maximal closed subgroup. Since $ G $ is finite then $ G $ is a compact subgroup of $ \G $ so must be contained in a maximal compact subgroup, call it $ \K $. But $ G $ is a maximal closed subgroup thus we must have that $ \K=\G $ (note that $ \K $ cannot equal $ G $ since $ \K $ is connected (the maximal compact of a connected group is always connected)). So $ \G $ must be compact. If $ \G $ is not simple then there exists some morphism
$$
\pi: \G \to \G_i
$$
with positive dimensional kernel (here $ \G_i $ is basically one of the semisimple factors of $ \G $). Then
$$
\pi^{-1}(\pi(G))
$$
is a closed positive dimensional subgroup containing $ G $, contradicting the fact that $ G $ is a finite maximal closed subgroup. Thus if a connected Lie group $ \G $ has a finite maximal closed subgroup then we can conclude that $ \G $ is simple.
However the reverse implication does not hold: $ SU_{15} $ is an example of a compact connected simple Lie group with no finite maximal closed subgroups.
To see why this is the case it is important to note that
Claim 2: For a compact connected simple Lie group $ \G $, $ G $ is a finite maximal closed subgroup of $ \G $ if and only if $ G $ is Ad-irreducible and $ G $ is a maximal finite subgroup of $ \G $.
this follows from Corollary 3.5 of Sawicki and Karnas - Universality of single qudit gates.
Since a finite subgroup of $ SU_n $ is Ad-irreducible if and only if it is a unitary 2-design we have
Claim 3: $ G $ is a finite maximal closed subgroup of $ SU_n $ if and only if $ G $ is a maximal unitary 2-group in $ SU_n $.
By inspecting Theorem 3 of Bannai, Navarro, Rizo, and Pham Huu Tiep - Unitary $t$-groups one immediately determines that $ SU_{15} $ has no finite maximal closed subgroups.
Some of the main examples of finite maximal closed subgroups of $ SU_n $ include the normalizer in $ SU_{p^n} $ of an extra-special group $ p^{2n+1} $. Here $ p $ is an odd prime. There is also a similar construction $ p=2 $. These are known as (complex) Clifford groups. Then there are infinite families of examples relating to the Weil module for $ \operatorname{PSp}_{2n}(3) $ and another family related to $ U_n(2) $. Plus many exceptional cases.
A similar normalizer construction to the above gives finite maximal closed subgroups of all the $ \operatorname{SO}(2^n) $ as normalizers of an extra-special group $ 2^{2n+1} $. This is known as the real Clifford group. For details about real and complex Clifford groups see Nebe, Rains, and Sloane - Self-Dual Codes and Invariant Theory.
