# Existence of proper invariant subset in an irreducible action

Let $G<\rm{GL}_n(\mathbb{k})$ be a linear group, where $\mathbb{k}$ is an algebraically closed field. Assume that the linear action of $G$ on $\mathbb{k}^n$ is strongly-irreducible (i.e. there are no $H$-invariant proper subspaces of $\mathbb{k}^n$, except $0$, for any $H< G$ of finite index). Equip $\mathbb{k}^n$ with the Zariski topology. Could there be a proper non-empty open subset $U\subset\mathbb{k}^n$ which is $G$-invariant, and $\mathbb{k}^n\setminus U\neq \{0\}$?

Thanks for any help.

-
In case $G$ is an algebraic group, your condition on subgroups of finite index will be taken care of most easily if $G$ is connected in the Zariski topology. (The answers given don't emphasize that part of the question.) –  Jim Humphreys May 5 '11 at 13:03

Let $Q$ be a non-degenerate quadratic form on $\mathbb{K}^n$, and $G=O(Q)$ its orthogonal group. I think that the set $U$ of vectors $x\in\mathbb{K}^n$ with $Q(x)\neq 0$ does the job.

-

Here are some more examples in the spirit of that of Alain Valette: let $G$ be a quasi-simple algebraic group and consider its adjoint representation on $\mathfrak{g} = \operatorname{Lie}(G)$. Then most of the time $\mathfrak{g}$ is a simple $G$-module (e.g. it is simple if the characteristic of $k$ is "very good" for $G$, and in particular, if the characteristic of $k$ is 0). And the action of $G$ leaves invariant the (closed) nilpotent subvariety $\mathcal{N}$ of $\mathfrak{g}$, and hence also its (open) complement $U = \mathfrak{g} \setminus \mathcal{N}$.

The adjoint action of $G$ on $\mathfrak{g}$ also leaves invariant these proper open sets of $\mathfrak{g}$:

$\bullet$ the regular elements $\mathfrak{g}_{\operatorname{reg}}$ (i.e. those elements with minimal dimensional centralizer), and

$\bullet$ the regular semisimple elements $\mathfrak{g}_{\operatorname{rs}}$ (those semisimple elements whose connected centralizer is a maximal torus).

-
@Alain Valette and George McNinch: Thanks a lot! your answers are very helpful! –  Dennis Gulko May 5 '11 at 11:04