In dimension $n=5$, does a subgroup of $O(n)$ satisfying these properties exist?

Question

I asked a question where @YCor provided a construction that seems to enable a group construction satisfying some properties when $n\ne 5$. However, in the case $n=5$, I am starting to think no such group exists. Precisely, I am asking if the following claim is true.

There is no closed subgroup $G$ of the the orthogonal group $O(5)$ along with a continuous surjective homomorphism $\phi : G\to \mathbb{Z}_2 = \{1, -1\}$ such that

1. If $x\in\mathbb{R}^n,\, x\ne 0$ then the orbit $Gx$ contains infinitely many distinct elements.
2. There exists $x\in \mathbb{R^n}$ such that $\{g\in G : gx = x\}\subseteq \operatorname{ker}\phi$.

Any ideas on how to approach this are appreciated.

Answer 1

I indeed think it doesn't exist for $n=5$.

By contradiction, let $G$ be such a subgroup, and $G^\circ$ its unit component. Decompose $\mathbf{R}^5$ as sum of irreducibles under $G^\circ$. By axiom 1, there is no 1-dimensional irreducible occurring in this decomposition. So the dimensions are either $5$ (irreducible action) or $3+2$.

In case it's $3+2$, $G^0$ has to be $\mathrm{SO}(3)\times\mathrm{SO}(2)$ under the expected action. In this case the normalizer of $G^0$ is $L=\mathrm{O}(3)\times\mathrm{O}(2)$, which contains $G^0$ with index $4$ and there are 4 candidates for $G$: $L$ and its three subgroups of index $2$: $L_1=\{(g,h)\in L:\det h=1\}$; $L_2=\{(g,h)\in L:\det(g)=1\}$, $L_3=\{(g,h)\in L:\det(g)=\det(h)\}$.

If we have $(x,y)\in\mathbf{R}^3\times\mathbf{R}^2$, let $r$ be a reflection of $\mathbf{R}^3$ fixing $x$ and let $r'$ be a reflection of $\mathbf{R}^2$ fixing $y$. Then at least one of $(\mathrm{id},r')$, $(r,\mathrm{id})$, $(r,r')$ belongs to $G\smallsetminus\mathrm{Ker}(\phi)$. So (2) fails.

In case the decomposition is "$5$", i.e., it's irreducible, then we can view the action as follows: let $Q$ be the 6-dimensional space of quadratic forms on $\mathbf{R}^3$; $\mathrm{GL}_3(\mathbf{R})$ naturally act on it. By definition, $\mathrm{O}(3)$ is the stabilizer of the quadratic form $q=x^2+y^2+z^2\in Q$. The action of $\mathrm{O}(3)$ on $Q$ preserves a unique hyperplane $H$ and $Q=H\oplus \mathbf{R}q$.

The normalizer of $\mathrm{SO}(3)$ in $\mathrm{O}(5)$ is then reduced to this given copy of $\mathrm{O}(3)$, so this is the only candidate for $G$ and $\phi$ has to be given by the determinant.

If $q'$ is any another quadratic form, there is an orthonormal basis on which $q'$ can be put in the form $ax^2+by^2+xz^2$. We then see that each diagonal matrix (under this basis) preserves $q'$. Thus $q'$ is stabilized by some element not in $\mathrm{Ker}(\phi)$. So (2) fails.