Openness of strong irreducibility

Let $$\Gamma$$ be a finitely generated group. A linear representation of $$\Gamma$$ is irreducible if it does not preserve a proper subspace, and strongly irreducible if it does not preserve a finite union of proper subspaces.

It is known that the set of irreducible representations is Zariski open in the variety of representations $$\mathrm{Hom}(\Gamma, SL(n,\mathbb{C}))$$ (see Sikora "Character varieties", prop. 27).

Is the set of strongly irreducible representations open in $$\mathrm{Hom}(\Gamma, SL(n,\mathbb{C}))$$, for either the Zariski or the classical topology ?

Initially, I thought it shouldn't be too hard to disprove Zariski openness even when $$\Gamma = F_2$$ and $$n = 2$$, using the fact that points of finite order are Zariski dense in $$\mathrm{SL}_n(\mathbf{C})$$ and representations of finite groups are never absolutely irreducible. This idea works for the analogous question in positive characteristic, since $$\mathrm{SL}_n(\overline{\mathbf{F}}_p)$$ is Zariski dense in the scheme $$\mathrm{SL}_{n, \overline{\mathbf{F}}_p}$$ and every finitely generated subgroup of $$\mathrm{SL}_n(\overline{\mathbf{F}}_p)$$ is finite.

However, to my surprise, the absolutely irreducible locus is always Zariski open (in characteristic 0)! The condition of absolute irreducibility makes sense if $$\mathrm{SL}_n$$ is replaced by any reductive group $$G$$ in characteristic 0, and we have the following result.

Theorem. Let $$k$$ be a field of characteristic 0. Let $$\mathcal{H}_0$$ be the subset of the $$k$$-scheme $$\mathrm{Hom}(\Gamma, G)$$ consisting of the absolutely irreducible representations. Then $$\mathcal{H}_0$$ is open.

To avoid uninteresting circumlocutions in the proof arising from the center of $$G$$ and $$\pi_0(G)$$, we will assume that $$G$$ is connected and semisimple. Even when $$G = \mathrm{SL}_n$$, the proof will involve studying essentially arbitrary (disconnected) reductive groups, and this justifies the generality. However, the proof simplifies slightly in this case, as I will indicate below.

Let $$\mathcal{H}$$ be the (open) irreducible locus in $$\mathrm{Hom}(\Gamma, G)$$, consisting (by Propositions 8 and 15 of Sikora's paper) of those $$\rho$$ such that $$H_\rho = \overline{\rho(\Gamma)}^0$$ (connected component of the Zariski closure of the image of $$\rho$$) is reductive and $$Z_G(\rho(\Gamma))$$ is finite. We wish to show that $$\mathcal{H}_0$$ is open in $$\mathcal{H}$$. We are going to try to understand $$\mathcal{H} - \mathcal{H}_0$$.

The proof is a bit long, so let me just explain the case $$G = \mathrm{SL}_2$$ first. Note that, if $$\rho \in \mathcal{H}$$, then $$H_\rho$$ is either trivial, a maximal torus $$T$$, or all of $$\mathrm{SL}_2$$ (the only absolutely irreducible case). It is well known that the only finite subgroups of $$\mathrm{PGL}_2(\mathbf{C})$$ are either cyclic, dihedral, or isomorphic to $$A_4$$, $$S_4$$, or $$A_5$$, and there is a unique conjugacy class of such subgroups in each case. In either of the first two cases, the subgroup normalizes a maximal torus. Thus $$\mathcal{H}_0$$ is the complement in $$\mathcal{H}$$ of the $$\mathrm{SL}_2(\mathbf{C})$$-orbits of $$\mathrm{Hom}(\Gamma, N_{\mathrm{SL}_2}(T))$$ and $$\bigcup_{i=1}^3 \mathrm{Hom}(\Gamma, \pi^{-1}(\Lambda_i))$$, where $$\pi\colon \mathrm{SL}_2 \to \mathrm{PGL}_2$$ is the quotient and the $$\Lambda_i$$ are $$A_4, S_4, A_5$$. Simple arguments show that each of these four orbits is Zariski closed in $$\mathcal{H}$$ (although the first is typically not closed in all of $$\mathrm{Hom}(\Gamma, \mathrm{SL}_2)$$), so $$\mathcal{H}_0$$ is open.

The key point above was that there are only finitely many conjugacy classes of finite subgroups of $$\mathrm{SL}_2(\mathbf{C})$$ not normalizing a maximal torus. The general argument is similar, but it is complicated by the fact that there are many more possibilities for $$H_{\rho}$$. We begin with two observations.

Observation 1. A representation $$\rho\colon \Gamma \to G(k)$$ is absolutely irreducible if and only if $$H_\rho = \overline{\rho(\Gamma)}^0 \subset G$$ is irreducible.

Proof. It is elementary to check that if $$\Gamma' \subset \Gamma$$ is a finite index subgroup, then $$H_\rho = H_{\rho|_{\Gamma'}}$$. Moreover, there exists some such $$\Gamma'$$ with the property that $$\rho(\Gamma') \subset H_\rho$$. Thus this follows from the definitions and the fact that $$\rho$$ is irreducible if and only if $$\overline{\rho(\Gamma)}$$ is irreducible. $$\square$$

Observation 2. If $$\rho$$ is absolutely irreducible, then the centralizer $$Z_G(H_\rho)$$ is finite, and in particular $$H_\rho$$ is semisimple. (The converse is also true in characteristic 0.)

Proof. Note that if the conclusion fails, then there is a nontrivial torus $$S$$ in the reductive group $$Z_G(H_\rho)$$. Then $$H_\rho$$ lies in $$Z_G(S)$$, a proper Levi subgroup of $$G$$ since $$G$$ is semisimple, and thus $$H_\rho$$ cannot be irreducible. Apply Observation 1. $$\square$$

By these observations, note that if $$\rho \in \mathcal{H}(k) - \mathcal{H}_0$$, then either

1. There is a nontrivial central torus of $$Z_G(H_\rho)$$, or
2. $$H_\rho$$ and $$Z_G(H_\rho)$$ are semisimple, and $$Z_G(H_\rho)$$ is positive-dimensional.

Thus we want to understand the locus of $$\rho$$ in $$\mathcal{H}$$ satisfying either 1 or 2. We understand these conditions in turns.

In case 1, let $$S$$ be the maximal central torus of $$Z_G(H_\rho)$$, and let $$L = Z_G(S)$$. Since $$\rho(\Gamma)$$ normalizes $$H_\rho$$, it must normalize $$Z_G(H_\rho)$$ and hence $$S$$. Thus $$\rho$$ lies in the image of \begin{align*} G \times \mathrm{Hom}(\Gamma, &N_G(L)) \to \mathrm{Hom}(\Gamma, G) \\ &(g, \rho_0) \mapsto g\rho_0 g^{-1}. \end{align*} Conversely, if $$\rho$$ lies in this image, then 1 holds.

Now we deal with case 2, which is the more interesting (and complicated) one. The main problem is that, although $$H_\rho$$ lies in a proper Levi $$L$$ of $$G$$, it is not typically true that $$\rho(\Gamma)$$ normalizes any such $$L$$. (Think about the case that $$G = \mathrm{SL}_n$$ and $$\rho$$ is isotypic.)

Proposition 1. If $$k$$ is of characteristic 0, there are only finitely many conjugacy classes of connected semisimple subgroups of $$G$$.

Proof. By the classification of reductive groups by root data, there are only finitely many isomorphism classes of semisimple groups $$H$$ of dimension $$\leq \operatorname{dim} G$$. Thus we fix an $$H$$ and show that there are only finitely many conjugacy classes of subgroups of $$G$$ which are isomorphic to $$H$$. If $$G = \mathrm{SL}_n$$, then this follows from the fact that every connected semisimple group has only finitely many isomorphism classes of representations of a fixed dimension. (Use the Weyl character formula.)

For general $$G$$, here is a high-powered argument: by SGA3, Exposé XXIV, Proposition 7.3.1, there is a smooth affine scheme $$\mathrm{Hom}(H, G)$$ which parameterizes homomorphisms $$f\colon H \to G$$. For such $$f$$, there is an orbit map $$\alpha_f\colon G \to \mathrm{Hom}(H, G)$$ given by $$g \mapsto gfg^{-1}$$. By SGA3, Exposé III, Théorème 2.1(ii), the obstruction to smoothness of $$\alpha_f$$ lies in $$\mathrm{H}^1(H, \mathfrak{g})$$, which vanishes because $$H$$ is reductive and we are in characteristic 0. Thus $$\alpha_f$$ is smooth, and in particular it has open image. Thus every orbit is open, from which it follows that every orbit is also closed. Being affine, it follows that $$\mathrm{Hom}(H, G)$$ is a finite union of clopen $$G$$-orbits. For each such orbit consisting of injective homomorphisms, choose a point $$\iota$$, and note that $$\{\iota(H)\}$$ constitutes a system of representatives for the subgroups of $$G$$ isomorphic to $$H$$. $$\square$$

Proposition 2. If $$k$$ is of characteristic $$0$$ and $$H$$ is a reductive group over $$k$$, then there is a positive integer $$N$$ such that for every finite subgroup $$\Lambda \subset H(k)$$ of order $$\geq N$$, there is a nontrivial torus $$S \subset H^0$$ normalized by $$\Lambda$$.

Proof. If $$H^0$$ has a nontrivial central torus then this is clear, so assume $$H^0$$ is semisimple. By Jordan's theorem and finiteness of $$H(k)/H^0(k)$$, there is a constant $$N_0$$ such that every finite subgroup $$\Lambda \subset H(k)$$ admits an abelian normal subgroup $$\Lambda_0 \subset \Lambda \cap H^0(k)$$ of index $$\leq N_0$$. Then $$\Lambda$$ normalizes $$Z_H(\Lambda_0)^0$$, so it suffices to show that if $$H$$ is connected then $$Z_H(\Lambda_0)^0$$ contains a nontrivial central torus for all sufficiently large finite abelian groups $$\Lambda_0 \subset H(k)$$. If $$H = \mathrm{SL}_n$$, then this is true for all $$\Lambda_0$$ of order $$\geq n$$ since all abelian groups can be diagonalized and the center is of order $$n$$, but in the application below, even when $$G = \mathrm{SL}_n$$, typically $$H^0$$ will be a nontrivial quotient of a centralizer in $$\mathrm{SL}_n$$; in particular, some nonobvious bound on the size of $$\Lambda_0$$ is really necessary, as even $$\mathrm{PGL}_2$$ contains an abelian subgroup of order $$4$$ which is its own centralizer. This phenomenon can occur even for simply connected groups in other types.

From now on, assume $$H$$ is connected. There is a constant $$N_1$$ such that if $$t \in H(k)$$ is semisimple then $$t^{N_1}$$ is contained in a nontrivial central torus of $$Z_H(t)^0$$: one way to see this is that, by Borel--de Siebenthal, if $$H'$$ is a maximal rank connected reductive subgroup of $$H$$ then $$H' = Z_H(Z(H'))$$. Since such $$H'$$ are determined by root subsystems of the root system for $$H$$, there are only finitely many conjugacy classes of these. Let $$N_1$$ be the maximum of $$|Z(H')/Z(H')^0|$$ where $$H'$$ ranges over the maximal rank connected reductive subgroups of $$H$$. If $$t \in H(k)$$ is semisimple, then since $$t$$ lies in $$Z(Z_H(t)^0)$$ and $$N_1$$ kills the component group of this center, we find that $$t^{N_1}$$ lies in the maximal central torus of $$Z_H(t)^0$$.

Let $$T$$ be a maximal torus of $$H$$, and let $$W$$ be the Weyl group of $$(H, T)$$. Let $$N_2$$ be the maximum of $$|S^{W'}/(S^{W'})^0| + 1$$, where $$S$$ ranges over all maximal central subtori of maximal rank connected reductive subgroups $$M$$ of $$H$$ containing $$S$$ and $$W'$$ ranges over all subgroups of $$W$$ which stabilize $$S$$. Let $$\Lambda_0 \subset H(k)$$ be a finite abelian subgroup containing an element $$t$$ of order $$\geq N_1 N_2$$. Let $$S$$ be the maximal central torus of $$Z_H(t)^0$$, which is nontrivial since it contains $$t^{N_1}$$ by the previous paragraph. Note that $$\Lambda_0$$ stabilizes $$S$$, and $$|S^{\Lambda_0}/(S^{\Lambda_0})^0| < N_2$$ by the above since $$N_G(S)/Z_G(S)$$ is a subquotient of $$W$$. Because $$S^{\Lambda_0}(k)$$ contains the order $$\geq N_2$$ element $$t^{N_1}$$, it follows that $$(S^{\Lambda_0})^0$$ is nontrivial. This exhibits a nontrivial central torus of $$Z_H(\Lambda_0)$$.

If $$H$$ embeds into $$\mathrm{GL}_{N_3}$$, then for any finite abelian subgroup $$\Lambda_0$$ of $$H(k)$$ of order $$\geq (N_1 N_2)^{N_3}$$ there is an element $$t \in \Lambda_0$$ of order $$\geq N_1 N_2$$ (since finite abelian subgroups of $$\mathrm{GL}_{N_3}$$ are diagonalizable). By the previous paragraph, $$Z_H(\Lambda_0)$$ contains a nontrivial central torus. $$\square$$

Proof of Theorem. By Proposition 1, there are connected semisimple subgroups $$H_1, \dots, H_m$$ of $$G$$ such that $$Z_G(H_i)$$ is positive-dimensional for all $$i$$ and such that if $$H \subset G$$ is any subgroup with these properties then $$H$$ is $$G(k)$$-conjugate to some such $$H_i$$. If $$\rho \in \mathcal{H}(k)$$ satisfies 2 above, then a $$G(k)$$-conjugate of $$\rho$$ factors through $$N_G(H_i)$$ for some $$i$$, and the induced map $$\Gamma \to N_G(H_i)(k)/H_i(k)$$ has finite image. If $$N$$ is as in Proposition 2 and this image is of order $$\geq N$$, then there is a nontrivial torus $$S' \subset N_G(H_i)/H_i$$ which is normalized by $$\Gamma$$. Since $$Z_G(H_i)$$ is semisimple by assumption, the map $$Z_G(H_i) \to N_G(H_i)^0/H_i$$ is a central isogeny. If $$S$$ is the identity component of the preimage of $$S'$$ in $$Z_G(H_i)$$, then $$S$$ is a torus which is also normalized by $$\rho(\Gamma)$$, so $$\rho(\Gamma)$$ factors through $$N_G(Z_G(S))$$.

Note that (a more elementary version of) the proof of Proposition 1 also shows that $$N_G(H_i)/H_i$$ has only finitely many conjugacy classes of finite subgroups of order $$< N$$, say $$\Gamma_{i1}, \dots, \Gamma_{ip_i}$$. For each pair $$(i, j)$$, let $$H_{ij}$$ be the preimage of $$\Gamma_{ij}$$ in $$N_G(H_i)$$. We just saw that if $$\rho$$ factors through $$N_G(H_i)$$, then either $$\rho$$ factors through the normalizer of a Levi or there is an $$N_G(H_i)(k)$$-conjugate of $$\rho$$ which factors through $$H_{ij}$$. Conversely, if $$\rho$$ factors through some $$H_{ij}$$ then $$H_\rho$$ is contained in $$H_{ij}^0 = H_i$$ and thus $$\rho$$ is in case 2.

The discussion above shows that $$$$\mathcal{H} - \mathcal{H}_0 = \mathcal{H} \cap G \cdot \left(\bigcup_L \mathrm{Hom}(\Gamma, N_G(L)) \cup \bigcup_{\substack{1 \leq i \leq m \\ 1 \leq j \leq p_i}} \mathrm{Hom}(\Gamma, H_{ij})\right),$$$$ where $$L$$ ranges over a (finite) set of representatives for the conjugacy classes of Levi subgroups of $$G$$. If $$H$$ is any reductive subgroup of $$G$$, then $$G \cdot \mathrm{Hom}(\Gamma, H)$$ is the image of the action morphism $$G \times \mathrm{Hom}(\Gamma, H) \to \mathrm{Hom}(\Gamma, G)$$, which is constructible by Chevalley's theorem. The displayed equation shows that $$\mathcal{H} - \mathcal{H}_0$$ is constructible, and thus $$\mathcal{H}_0$$ is constructible.

If $$\Gamma'$$ is a subgroup of $$\Gamma$$, let $$\pi_{\Gamma'}\colon \mathrm{Hom}(\Gamma, G) \to \mathrm{Hom}(\Gamma', G)$$ be the restriction map. Then $$\mathcal{H}_0$$ is the intersection of $$\{\mathcal{H}_{\Gamma'} = \pi_{\Gamma'}^{-1}(\{\text{irreducible representations}\})\}$$, where $$\Gamma'$$ ranges over all finite index subgroups of $$\Gamma$$. Each $$\mathcal{H}_{\Gamma'}$$ is open in the scheme $$\mathrm{Hom}(\Gamma, G)$$, hence closed under generization, and this property is clearly stable under intersections. Since a constructible subset of a noetherian scheme is open if and only if it is closed under generization, we are done. $$\square$$

• Thank you for such a complete answer ! I am slowly processing it, I'll get back to you soon.
– FMB
Commented May 14 at 16:30