Initially, I thought it shouldn't be too hard to disprove Zariski openness even when $\Gamma = F_2$ and $n = 2$, using the fact that points of finite order are Zariski dense in $\mathrm{SL}_n(\mathbf{C})$ and representations of finite groups are never absolutely irreducible. This idea works for the analogous question in positive characteristic, since $\mathrm{SL}_n(\overline{\mathbf{F}}_p)$ is Zariski dense in the scheme $\mathrm{SL}_{n, \overline{\mathbf{F}}_p}$ and every finitely generated subgroup of $\mathrm{SL}_n(\overline{\mathbf{F}}_p)$ is finite.
However, to my surprise, the absolutely irreducible locus is always Zariski open (in characteristic 0)! The condition of absolute irreducibility makes sense if $\mathrm{SL}_n$ is replaced by any reductive group $G$ in characteristic 0, and we have the following result.
Theorem. Let $k$ be a field of characteristic 0. Let $\mathcal{H}_0$ be the subset of the $k$-scheme $\mathrm{Hom}(\Gamma, G)$ consisting of the absolutely irreducible representations. Then $\mathcal{H}_0$ is open.
To avoid uninteresting circumlocutions in the proof arising from the center of $G$ and $\pi_0(G)$, we will assume that $G$ is connected and semisimple. Even when $G = \mathrm{SL}_n$, the proof will involve studying essentially arbitrary (disconnected) reductive groups, and this justifies the generality. However, the proof simplifies slightly in this case, as I will indicate below.
Let $\mathcal{H}$ be the (open) irreducible locus in $\mathrm{Hom}(\Gamma, G)$, consisting (by Propositions 8 and 15 of Sikora's paper) of those $\rho$ such that $H_\rho = \overline{\rho(\Gamma)}^0$ (connected component of the Zariski closure of the image of $\rho$) is reductive and $Z_G(\rho(\Gamma))$ is finite. We wish to show that $\mathcal{H}_0$ is open in $\mathcal{H}$. We are going to try to understand $\mathcal{H} - \mathcal{H}_0$.
The proof is a bit long, so let me just explain the case $G = \mathrm{SL}_2$ first. Note that, if $\rho \in \mathcal{H}$, then $H_\rho$ is either trivial, a maximal torus $T$, or all of $\mathrm{SL}_2$ (the only absolutely irreducible case). It is well known that the only finite subgroups of $\mathrm{PGL}_2(\mathbf{C})$ are either cyclic, dihedral, or isomorphic to $A_4$, $S_4$, or $A_5$, and there is a unique conjugacy class of such subgroups in each case. In either of the first two cases, the subgroup normalizes a maximal torus. Thus $\mathcal{H}_0$ is the complement in $\mathcal{H}$ of the $\mathrm{SL}_2(\mathbf{C})$-orbits of $\mathrm{Hom}(\Gamma, N_{\mathrm{SL}_2}(T))$ and $\bigcup_{i=1}^3 \mathrm{Hom}(\Gamma, \pi^{-1}(\Lambda_i))$, where $\pi\colon \mathrm{SL}_2 \to \mathrm{PGL}_2$ is the quotient and the $\Lambda_i$ are $A_4, S_4, A_5$. Simple arguments show that each of these four orbits is Zariski closed in $\mathcal{H}$ (although the first is typically not closed in all of $\mathrm{Hom}(\Gamma, \mathrm{SL}_2)$), so $\mathcal{H}_0$ is open.
The key point above was that there are only finitely many conjugacy classes of finite subgroups of $\mathrm{SL}_2(\mathbf{C})$ not normalizing a maximal torus. The general argument is similar, but it is complicated by the fact that there are many more possibilities for $H_{\rho}$. We begin with two observations.
Observation 1. A representation $\rho\colon \Gamma \to G(k)$ is absolutely irreducible if and only if $H_\rho = \overline{\rho(\Gamma)}^0 \subset G$ is irreducible.
Proof. It is elementary to check that if $\Gamma' \subset \Gamma$ is a finite index subgroup, then $H_\rho = H_{\rho|_{\Gamma'}}$. Moreover, there exists some such $\Gamma'$ with the property that $\rho(\Gamma') \subset H_\rho$. Thus this follows from the definitions and the fact that $\rho$ is irreducible if and only if $\overline{\rho(\Gamma)}$ is irreducible. $\square$
Observation 2. If $\rho$ is absolutely irreducible, then the centralizer $Z_G(H_\rho)$ is finite, and in particular $H_\rho$ is semisimple. (The converse is also true in characteristic 0.)
Proof. Note that if the conclusion fails, then there is a nontrivial torus $S$ in the reductive group $Z_G(H_\rho)$. Then $H_\rho$ lies in $Z_G(S)$, a proper Levi subgroup of $G$ since $G$ is semisimple, and thus $H_\rho$ cannot be irreducible. Apply Observation 1. $\square$
By these observations, note that if $\rho \in \mathcal{H}(k) - \mathcal{H}_0$, then either
- There is a nontrivial central torus of $Z_G(H_\rho)$, or
- $H_\rho$ and $Z_G(H_\rho)$ are semisimple, and $Z_G(H_\rho)$ is positive-dimensional.
Thus we want to understand the locus of $\rho$ in $\mathcal{H}$ satisfying either 1 or 2. We understand these conditions in turns.
In case 1, let $S$ be the maximal central torus of $Z_G(H_\rho)$, and let $L = Z_G(S)$. Since $\rho(\Gamma)$ normalizes $H_\rho$, it must normalize $Z_G(H_\rho)$ and hence $S$. Thus $\rho$ lies in the image of
\begin{align*}
G \times \mathrm{Hom}(\Gamma, &N_G(L)) \to \mathrm{Hom}(\Gamma, G) \\
&(g, \rho_0) \mapsto g\rho_0 g^{-1}.
\end{align*}
Conversely, if $\rho$ lies in this image, then 1 holds.
Now we deal with case 2, which is the more interesting (and complicated) one. The main problem is that, although $H_\rho$ lies in a proper Levi $L$ of $G$, it is not typically true that $\rho(\Gamma)$ normalizes any such $L$. (Think about the case that $G = \mathrm{SL}_n$ and $\rho$ is isotypic.)
Proposition 1. If $k$ is of characteristic 0, there are only finitely many conjugacy classes of connected semisimple subgroups of $G$.
Proof. By the classification of reductive groups by root data, there are only finitely many isomorphism classes of semisimple groups $H$ of dimension $\leq \operatorname{dim} G$. Thus we fix an $H$ and show that there are only finitely many conjugacy classes of subgroups of $G$ which are isomorphic to $H$. If $G = \mathrm{SL}_n$, then this follows from the fact that every connected semisimple group has only finitely many isomorphism classes of representations of a fixed dimension. (Use the Weyl character formula.)
For general $G$, here is a high-powered argument: by SGA3, Exposé XXIV, Proposition 7.3.1, there is a smooth affine scheme $\mathrm{Hom}(H, G)$ which parameterizes homomorphisms $f\colon H \to G$. For such $f$, there is an orbit map $\alpha_f\colon G \to \mathrm{Hom}(H, G)$ given by $g \mapsto gfg^{-1}$. By SGA3, Exposé III, Théorème 2.1(ii), the obstruction to smoothness of $\alpha_f$ lies in $\mathrm{H}^1(H, \mathfrak{g})$, which vanishes because $H$ is reductive and we are in characteristic 0. Thus $\alpha_f$ is smooth, and in particular it has open image. Thus every orbit is open, from which it follows that every orbit is also closed. Being affine, it follows that $\mathrm{Hom}(H, G)$ is a finite union of clopen $G$-orbits. For each such orbit consisting of injective homomorphisms, choose a point $\iota$, and note that $\{\iota(H)\}$ constitutes a system of representatives for the subgroups of $G$ isomorphic to $H$. $\square$
Proposition 2. If $k$ is of characteristic $0$ and $H$ is a reductive group over $k$, then there is a positive integer $N$ such that for every finite subgroup $\Lambda \subset H(k)$ of order $\geq N$, there is a nontrivial torus $S \subset H^0$ normalized by $\Lambda$.
Proof. If $H^0$ has a nontrivial central torus then this is clear, so assume $H^0$ is semisimple. By Jordan's theorem and finiteness of $H(k)/H^0(k)$, there is a constant $N_0$ such that every finite subgroup $\Lambda \subset H(k)$ admits an abelian normal subgroup $\Lambda_0 \subset \Lambda \cap H^0(k)$ of index $\leq N_0$. Then $\Lambda$ normalizes $Z_H(\Lambda_0)^0$, so it suffices to show that if $H$ is connected then $Z_H(\Lambda_0)^0$ contains a nontrivial central torus for all sufficiently large finite abelian groups $\Lambda_0 \subset H(k)$. If $H = \mathrm{SL}_n$, then this is true for all $\Lambda_0$ of order $\geq n$ since all abelian groups can be diagonalized and the center is of order $n$, but in the application below, even when $G = \mathrm{SL}_n$, typically $H^0$ will be a nontrivial quotient of a centralizer in $\mathrm{SL}_n$; in particular, some nonobvious bound on the size of $\Lambda_0$ is really necessary, as even $\mathrm{PGL}_2$ contains an abelian subgroup of order $4$ which is its own centralizer. This phenomenon can occur even for simply connected groups in other types.
From now on, assume $H$ is connected. There is a constant $N_1$ such that if $t \in H(k)$ is semisimple then $t^{N_1}$ is contained in a nontrivial central torus of $Z_H(t)^0$: one way to see this is that, by Borel--de Siebenthal, if $H'$ is a maximal rank connected reductive subgroup of $H$ then $H' = Z_H(Z(H'))$. Since such $H'$ are determined by root subsystems of the root system for $H$, there are only finitely many conjugacy classes of these. Let $N_1$ be the maximum of $|Z(H')/Z(H')^0|$ where $H'$ ranges over the maximal rank connected reductive subgroups of $H$. If $t \in H(k)$ is semisimple, then since $t$ lies in $Z(Z_H(t)^0)$ and $N_1$ kills the component group of this center, we find that $t^{N_1}$ lies in the maximal central torus of $Z_H(t)^0$.
Let $T$ be a maximal torus of $H$, and let $W$ be the Weyl group of $(H, T)$. Let $N_2$ be the maximum of $|S^{W'}/(S^{W'})^0| + 1$, where $S$ ranges over all maximal central subtori of maximal rank connected reductive subgroups $M$ of $H$ containing $S$ and $W'$ ranges over all subgroups of $W$ which stabilize $S$. Let $\Lambda_0 \subset H(k)$ be a finite abelian subgroup containing an element $t$ of order $\geq N_1 N_2$. Let $S$ be the maximal central torus of $Z_H(t)^0$, which is nontrivial since it contains $t^{N_1}$ by the previous paragraph. Note that $\Lambda_0$ stabilizes $S$, and $|S^{\Lambda_0}/(S^{\Lambda_0})^0| < N_2$ by the above since $N_G(S)/Z_G(S)$ is a subquotient of $W$. Because $S^{\Lambda_0}(k)$ contains the order $\geq N_2$ element $t^{N_1}$, it follows that $(S^{\Lambda_0})^0$ is nontrivial. This exhibits a nontrivial central torus of $Z_H(\Lambda_0)$.
If $H$ embeds into $\mathrm{GL}_{N_3}$, then for any finite abelian subgroup $\Lambda_0$ of $H(k)$ of order $\geq (N_1 N_2)^{N_3}$ there is an element $t \in \Lambda_0$ of order $\geq N_1 N_2$ (since finite abelian subgroups of $\mathrm{GL}_{N_3}$ are diagonalizable). By the previous paragraph, $Z_H(\Lambda_0)$ contains a nontrivial central torus. $\square$
Proof of Theorem. By Proposition 1, there are connected semisimple subgroups $H_1, \dots, H_m$ of $G$ such that $Z_G(H_i)$ is positive-dimensional for all $i$ and such that if $H \subset G$ is any subgroup with these properties then $H$ is $G(k)$-conjugate to some such $H_i$. If $\rho \in \mathcal{H}(k)$ satisfies 2 above, then a $G(k)$-conjugate of $\rho$ factors through $N_G(H_i)$ for some $i$, and the induced map $\Gamma \to N_G(H_i)(k)/H_i(k)$ has finite image. If $N$ is as in Proposition 2 and this image is of order $\geq N$, then there is a nontrivial torus $S' \subset N_G(H_i)/H_i$ which is normalized by $\Gamma$. Since $Z_G(H_i)$ is semisimple by assumption, the map $Z_G(H_i) \to N_G(H_i)^0/H_i$ is a central isogeny. If $S$ is the identity component of the preimage of $S'$ in $Z_G(H_i)$, then $S$ is a torus which is also normalized by $\rho(\Gamma)$, so $\rho(\Gamma)$ factors through $N_G(Z_G(S))$.
Note that (a more elementary version of) the proof of Proposition 1 also shows that $N_G(H_i)/H_i$ has only finitely many conjugacy classes of finite subgroups of order $< N$, say $\Gamma_{i1}, \dots, \Gamma_{ip_i}$. For each pair $(i, j)$, let $H_{ij}$ be the preimage of $\Gamma_{ij}$ in $N_G(H_i)$. We just saw that if $\rho$ factors through $N_G(H_i)$, then either $\rho$ factors through the normalizer of a Levi or there is an $N_G(H_i)(k)$-conjugate of $\rho$ which factors through $H_{ij}$. Conversely, if $\rho$ factors through some $H_{ij}$ then $H_\rho$ is contained in $H_{ij}^0 = H_i$ and thus $\rho$ is in case 2.
The discussion above shows that
\begin{equation}
\mathcal{H} - \mathcal{H}_0 = \mathcal{H} \cap G \cdot \left(\bigcup_L \mathrm{Hom}(\Gamma, N_G(L)) \cup \bigcup_{\substack{1 \leq i \leq m \\ 1 \leq j \leq p_i}} \mathrm{Hom}(\Gamma, H_{ij})\right),
\end{equation}
where $L$ ranges over a (finite) set of representatives for the conjugacy classes of Levi subgroups of $G$. If $H$ is any reductive subgroup of $G$, then $G \cdot \mathrm{Hom}(\Gamma, H)$ is the image of the action morphism $G \times \mathrm{Hom}(\Gamma, H) \to \mathrm{Hom}(\Gamma, G)$, which is constructible by Chevalley's theorem. The displayed equation shows that $\mathcal{H} - \mathcal{H}_0$ is constructible, and thus $\mathcal{H}_0$ is constructible.
If $\Gamma'$ is a subgroup of $\Gamma$, let $\pi_{\Gamma'}\colon \mathrm{Hom}(\Gamma, G) \to \mathrm{Hom}(\Gamma', G)$ be the restriction map. Then $\mathcal{H}_0$ is the intersection of $\{\mathcal{H}_{\Gamma'} = \pi_{\Gamma'}^{-1}(\{\text{irreducible representations}\})\}$, where $\Gamma'$ ranges over all finite index subgroups of $\Gamma$. Each $\mathcal{H}_{\Gamma'}$ is open in the scheme $\mathrm{Hom}(\Gamma, G)$, hence closed under generization, and this property is clearly stable under intersections. Since a constructible subset of a noetherian scheme is open if and only if it is closed under generization, we are done. $\square$
