Question

Let G be an linear algebraic group, and let H be a zariski dense subgroup of G. Then does H have to be a irreducible subgroup of G? Here H being irreducible means that H has no nontrivial invariant subspaces. Assume that G comes with a faithful irreducible representation. For example we can take G=SL(n,R) or SO(n,R). In these cases subgroup H being irrreducible means that when H acts on R^n there is no nontrivial invariant subspace.

Answer 1

If $G$ is a linear algebraic group, then you can recognize the coordinate algebra of $G$ as the algebra generated by all matrix entries of all algebraic representations of $G$. (In other words, every matrix entry is a function on $G$, they are all regular functions, and they generate the coordinate ring $O(G)$.) This implies that if $H \subseteq G$ is Zariski dense, then the $H$-decomposition of every algebraic representation of $G$ is exactly the same as the $G$-decomposition. Because, if $W \subseteq V$ is an $H$-subrepresentation, then this says that in a basis of $V$ that extends a basis of $W$, certain matrix entries vanish on elements of $H$; then since $H$ is Zariski dense, the same matrix entries vanish on all of $G$. In particular, every $G$-irreducible representation is also $H$-irreducible.

Answer 2

Let me add another perspective. $H = SL_2(\mathbb{Z})$ is Zariski dense in $G=SL_2(\mathbb{R})$, but the restriction of a holomorphic discrete series representation of $G$ to $H$ is never irreducible. This is OK because these are not algebraic reps of $G$. [1] is a good reference about this.

[1] B. Bekka. Square integrable representations, von Neumann algebras and an application to Gabor analysis. J. Fourier Anal. Appl., 10(4):325–349, 2004.