Return to Answer

Update. I here answer the question in the comment: yes, every f.g. finite-by-abelian group occurs. Every such group $P$ can be written as fibered product $P=F\times_\Lambda\Gamma$, where $\Gamma$ is a free abelian group of finite rank, $F$ is a finite group, and $\Lambda$ is a common quotient (thus a finite abelian group). Choose $n$ and an embedding $F\to S=\mathrm{SL}_n(\mathbf{C})$, and an embedding of $\Lambda$ into a complex torus $T$ (defining by composition a map $F\to T$). Now embed $F$ diagonally into $S\times T$. Then the fundamental group of $(S\times T)/F$ is isomorphic to $P$ (see Corollary 3.1 of this note of mine with Borovoi; possibly there are earlier references). Finally use the theorem you refer to so as to realize $F$ as stabilizer of some vector for some $(S\times T)$-module.

Actually, a central-by-finite-by-abelian f.g. group is either finite-by-abelian or contains a copy of the discrete Heisenberg group so in a sense it's the last guy to rule out (if what follows is correct).

By definition, we have $\mathrm{Ker}(p\circ j)\cap j^{-1}(H)=j^{-1}(L'\cap H)$, which equals $j^{-1}(L')\cap j^{-1}(H)$. Recall that $L'\cap H\subset L'\subset j(G)$, so its Lie algebra is the image by $j$ of some Lie subalgebra of the Lie algebra of $G$, generating an immersed subgroup $M$. Then $j(M)$ is the unit component of $L'\cap H$ (in the Lie topology), so $\mathrm{Ker}(j)M$ has finite index in $j^{-1}(L'\cap H)$.

Thus $j^{-1}(H)$ is an iterated extension of $\mathrm{Ker}(j)M$, a finite group, and a compactly generated abelian Lie group. Thus the group of components $\pi_0(j^{-1}(H))$ is extension of the central subgroup $\mathrm{Ker}(j)$ by a finite-by-abelian finitely generated group. [Since central-by-finite implies finite-by-abelian, this is also central-by-(2-step-nilpotent)].

Actually, a central-by-(finite-by-abelian) f.g. group is either finite-by-abelian or contains a copy of the discrete Heisenberg group so in a sense it's the last guy to rule out (if what follows is correct).

By definition, we have $\mathrm{Ker}(p\circ j)\cap j^{-1}(H)=j^{-1}(L'\cap H)$, which equals $j^{-1}(L')\cap j^{-1}(H)$. Recall that $L'\cap H\subset L'\subset j(G)$, so its Lie algebra is the image by $j$ of some Lie subalgebra of the Lie algebra of $G$, generating an immersed subgroup $M$, which is normal in $j^{-1}(H)$. Then $j(M)$ is the unit component of $L'\cap H$ (in the Lie topology), so $\mathrm{Ker}(j)M$ has finite index in $j^{-1}(L'\cap H)$.

To summarize, we have the inclusions $\mathrm{Ker}(j)M\subset j^{-1}(L'\cap H)\subset j(H)$; these are closed normal subgroups, $M$ is connected, $\mathrm{Ker}(j)$ is central, the quotient $j^{-1}(L'\cap H)/\mathrm{Ker}(j)M$ is finite and the quotient $j^{-1}(H)/j^{-1}(L'\cap H)$ is abelian. Let $p$ be the projection $j^{-1}(H)\to \Gamma=\pi_0(j^{-1}(H))$. Then $p(M)$ is trivial, so $p(\mathrm{Ker}(j)M)=p(\mathrm{Ker}(j))$ is central; the quotient $p(j^{-1}(L'\cap H))/p(\mathrm{Ker}(j)M)$ is finite and the quotient $p(j^{-1}(H))/p(j^{-1}(L'\cap H))$ is abelian. Thus $\Gamma$ is central-by-(finite-by-abelian). [Since central-by-finite implies finite-by-abelian, $\Gamma$ is also central-by-(2-step-nilpotent)].

First let me assume only that $G$ is a connected Lie group because it's more flexible. Then clearly the question does not change if we mod out by the unit component of the kernel of the representation, so we can suppose that the representation has a discrete (hence central) kernel.

So now we have: a connected Lie group $G$, a continuous homomorphism $j$ with discrete kernel and Zariski-dense image into a real connected linear algebraic group $L_a$, a Zariski-closed subgroup $H_a$ in $L_a$, and we wish to understand the group of components of $j^{-1}(H_a)$. The group of real points of $L_a$ has finitely many components in the Lie topology; let $L$ be the unit component, and let $H=L\cap H_a$. Since $j(G)\subset H$, we have $j^{-1}(H)=j^{-1}(H_a)$.

Since $j(G)$ is Zariski dense in $L$, its Lie algebra is an ideal in the Lie algebra of $L$ with abelian quotient. In particular (by Zariski connectedness of $L$) its Lie algebra is normalized by $L$, and thus by connectedness of $G$, we obtain that $j(G)$ is normal in $L$ with abelian quotient $(*)$. In particular, the derived subgroup $L'$ of $L$ (viewed as the group of real points) is contained in $j(G)$. (This derived subgroup has finite index in the group of real points of the algebraic group $L$.)

Let $p$ be the projection $L\to L/L'$. Then $L/L'$ is an abelian virtually connected Lie group; $p(H)$ is closed in $L/L'$; the image $p\circ j(G)\subset L/L'$ is a connected immersed Lie subgroup, and hence $(p\circ j)^{-1}(p(H))/\mathrm{Ker}(p\circ j)$, as a closed subgroup of a connected abelian Lie group, is isomorphic to the direct product of a connected abelian Lie group and a free abelian group of finite rank.

By definition, we have $\mathrm{Ker}(p\circ j)\cap j^{-1}(H)=j^{-1}(L'\cap H)$, which equals $j^{-1}(L')\cap j^{-1}(H)$. Recall that $L'\cap H\subset L'\subset j(G)$, so its Lie algebra is the image by $j$ of some Lie subalgebra of the Lie algebra of $G$, generating an immersed subgroup $M$. Then $j(M)$ is the unit component of $L'\cap H$ (in the Lie topology), so $j^{-1}(L'\cap H)$ has finite index in $\mathrm{Ker}(j)M$.

Let me assume only that $G$ is a connected Lie group because simple connectedness does not seem to play any role. Clearly the question does not change if we mod out by the unit component of the kernel of the representation, so we can suppose that the representation has a discrete (hence central) kernel.

So now we have: a connected Lie group $G$, a continuous homomorphism $j$ with discrete kernel and Zariski-dense image into a real connected linear algebraic group $L_a$, a Zariski-closed subgroup $H_a$ in $L_a$, and we wish to understand the group of components of $j^{-1}(H_a)$. The group of real points of $L_a$ has finitely many components in the Lie topology; let $L$ be the unit component, and let $H=L\cap H_a$. Since $j(G)\subset L$, we have $j^{-1}(H)=j^{-1}(H_a)$.

Since $j(G)$ is Zariski dense in $L$, its Lie algebra is an ideal in the Lie algebra of $L$ with abelian quotient. In particular (by Zariski connectedness of $L$) its Lie algebra is normalized by $L$, and thus by connectedness of $G$, we obtain that $j(G)$ is normal in $L$ with abelian quotient $(*)$. In particular, the derived subgroup $L'$ of $L$ (viewed as the group of real points) is contained in $j(G)$. (This derived subgroup has finite index in the group of real points of the algebraic group $[L_a,L_a]$.)

Let $p$ be the projection $L\to L/L'$. Then $L/L'$ is an abelian connected Lie group; $p(H)$ is closed in $L/L'$; the image $p\circ j(G)\subset L/L'$ is a connected immersed Lie subgroup, and hence $(p\circ j)^{-1}(p(H))/\mathrm{Ker}(p\circ j)$, as a closed subgroup of a connected abelian Lie group, is isomorphic to the direct product of a connected abelian Lie group and a free abelian group of finite rank.

By definition, we have $\mathrm{Ker}(p\circ j)\cap j^{-1}(H)=j^{-1}(L'\cap H)$, which equals $j^{-1}(L')\cap j^{-1}(H)$. Recall that $L'\cap H\subset L'\subset j(G)$, so its Lie algebra is the image by $j$ of some Lie subalgebra of the Lie algebra of $G$, generating an immersed subgroup $M$. Then $j(M)$ is the unit component of $L'\cap H$ (in the Lie topology), so $\mathrm{Ker}(j)M$ has finite index in $j^{-1}(L'\cap H)$.