Suppose that $G$ is a finite subgroup of ${\rm SO}(3)$. Is there a *smooth* self-map of ${\bf R}^3$ whose fibers are precisely the orbits of $G$?

Question

$\newcommand\R{\mathbf R}\DeclareMathOperator\SO{SO}\newcommand\C{\mathbf C}$The question is motivated by the theory of orbifolds. If $\mathcal O$ is an orientable $3$-orbifold (without boundary), an affirmative answer to the question would give a smooth manifold structure on the underlying space $X_{\mathcal O}$ with the property that all orbifold charts, regarded as maps from open subsets of $\R^3$ to $X_{\mathcal O}$, are smooth.

If one replaces $\SO(3)$ and $\R^3$ by $\SO(2)$ and $\R^2$, the answer is affirmative and easy: $G$ must be cyclic, and if we identify $\R^2$ with the field $\C$ of complex numbers, a generator acts by $z\mapsto\exp(2\pi i/n)z$ for some $n$. The required map is then $z\mapsto z^n$. This also gives an affirmative answer in the three-dimensional setting if one assumes $G$ is cyclic: if we identify $\R^3$ with $\C\times\R$, we may assume after conjugation that the generator acts by $(z,t)\mapsto(\exp(2\pi i/n)z,t)$ for some $n$. The required map is then $(z,t)\mapsto (z^n,t) $.

The answer is also affirmative if one asks for a continuous map instead of a smooth one. It follows by radial extension from the fact that the quotient of $S^2$ by a finite group, which acts by orientation-preserving homeomorphisms, is homeomorphic to $S^2$. This can even be used to get a continuous map having the desired property, and smooth except at the origin.

The interesting case is the one in which $G$ is non-cyclic, which means it is a spherical triangle group. It can be as "simple" as a dihedral group of arbitrary finite order, or as "complicated" as the alternating group on five letters.

Answer 1

Here is a standard trick: Let $f: S^{n-1}\to S^{n-1}$ be a smooth map. We would like to extend it "radially" to a smooth map $F: \mathbb R^{n}\to \mathbb R^{n}$ such that if $f$ is $\rho$-equivariant, so is $F$, where, $G< O(n)$ and $\rho: G\to O(n)$ is a representation (in your case, trivial representation). For $x\in \mathbb R^n\setminus \{0\}$ set $r:=|x|$ and $\bar x:= x/r$. Then for $x\in \mathbb R^n \setminus \{0\}$define $$ F(x)= e^{-1/r} f(\bar x), F(0)=0. $$
Clearly, $F$ is again $\rho$-equivariant. One verifies that $F$ is smooth: This is clear away from the origin and at the origin follows from vanishing of derivatives of all orders of the function $r\mapsto \exp(-1/r), r> 0$, at $r= 0$.

One uses the same argument to show that for every Milnor's exotic sphere $\Sigma^7$ there is a smooth homeomorphism $\Sigma^7\to S^7$.

Edit. Even though you already know how to construct the required smooth map $f: S^2\to S^2$ given a finite subgroup $G< SO(3)$, here is how I would do this. Let $\Sigma\subset S^2$ denote the complement to the set of fixed points of nontrivial elements of $G$ in $S^2$. The group $G$ acts freely (properly) and isometrically on $\Sigma$ (preserving the restriction of the standard Riemannian metric on $S^2$), hence, the quotient $\Sigma':=\Sigma/G$ has a Riemannian metric with respect to which the projection $\pi: \Sigma\to \Sigma'$ is a local isometry. Regarding $\Sigma'$ as a Riemann surface (with respect to the above Riemannian structure) and using the uniformization theorem, find a holomorphic embedding $h: \Sigma'\to S^2$. Riemann extension theorem implies that $h\circ \pi: \Sigma\to S^2$ extends to a holomorphic map $f: S^2\to S^2$. From the construction, it is clear that $f(x)=f(y)$ if and only if $Gx=Gy$. Thus, $f$ is the required smooth map $S^2\to S^2$. Now, apply the above extension construction and get a smooth map $$ F: \mathbb R^3\to \mathbb R^3 $$ which also has the property $F(x)=F(y)\iff Gx=Gy$. This gives a positive answer to your question.

Answer 2

$\def\RR{\mathbb{R}}\def\CC{\mathbb{C}}$I have been thinking about whether we could find a polynomial map with these properties. The answer is yes in every case except possibly one: I have checked the case of the cyclic groups, the dihedral groups, the rotational symmetries of the tetrahedron, and the rotational symmetries of the cube/octahedron. I have not been able to figure out whether such a polynomial map exists for the icosahedron/dodecahedron.

As a reminder, this is the full list of finite subgroups of $SO(3)$.

The cyclic case: Let $C_n$ be the cyclic group of order $n$. Identify $\RR^3$ with $\CC \times \RR$ and let $\zeta$ be a primitive $n$-th root of unity. Then $C_n$ acts on $\CC \times \RR$ by maps of the form $(w, t) \mapsto (\zeta^k w, t)$. So the map $(w,t) \mapsto (w^n, t)$ is a polynomial map whose fibers are the orbits of the cyclic group. Note that $\text{Re}((x+iy)^n)$, $\text{Im}((x+iy)^n)$ do NOT generate the ring of $C_n$-invariants inside $\RR[x,y]$: The polynomial $x^2+y^2$ is $C_n$-invariant, but not in the ring generated by $\text{Re}((x+iy)^n)$, $\text{Im}((x+iy)^n)$ for $n>2$.

The dihedral case: Let $D_n = C_n \rtimes C_2$. Again, identify $\RR^3$ with $\CC \times \RR$ and let $\zeta$ be a primitive $n$-th root of unity. In the dihedral action, the subgroup $C_n$ acts by $(w, t) \mapsto (\zeta^k w, t)$ and the nontrivial element of $C_2$ acts by $(w, t) \mapsto (\overline{w}, -t)$.

Put $w' = w^n$, so $w'$ and $t$ are invariant for the subgroup $C_n$ of $D_n$. Then $D_n/C_n$ acts by $(w', t) \mapsto (\overline{w'}, -t)$. Write $w' = x'+iy'$. Then the action of $D_n/C_n$ on $(x', y', t)$ is by $(x', y', t) \mapsto (x', -y', -t)$. We can think of this as a $C_2$ action as in the cyclic case, so we can compose the map $(x,y,t) \to (x', y', t)$ with the quotient by $C_2$ from the cyclic case to get a polynomial map whose fibers are the $D_n$-orbits. Explicitly, the three polynomials are $${\Big(} \text{Re}{\big(}(x+iy)^n{\big)},\ \text{Re}{\big(}(\text{Im}((x+iy)^n) + i t)^2{\big)},\ \text{Im}{\big(}(\text{Im}((x+iy)^n) + i t)^2{\big)}{\Big)}.$$

The Klein $4$-group, again Let $K$ be the Klein $4$-group $C_2 \times C_2$, acting by the diagonal $\pm 1$ matrices with determinant $1$. This is the case $n=2$ of the previous example, so we already have a solution in this case: Explicitly, the above computation gives $(x,y,z) \mapsto (x^2-y^2, 4 x^2 y^2 - z^2, 2 xyz)$ as a solution. However, we want a different solution to use as a building block for the tetrahedral and cubical cases. Thus, the purpose of this section is to verify that $$(x,y,z) \mapsto (x^2-y^2, y^2-z^2, xyz)$$ also works. It is clear that these polynomials are $K$-invariant, so what remains to show is that, given a triples of real numbers $(a,b,t)$, the set of real solutions to $x^2-y^2 = a$, $y^2-z^2 = b$, $xyz=t$ is precisely one $K$-orbit.

First of all, negating all of $(x,y,z)$ preserves $x^2-y^2$ and $y^2-z^2$, while negating $xyz$, so we may as well assume that $t \geq 0$. At which point, it is equivalent to show that there is a unique solution for $(x,y,z) \in \RR_{\geq 0}^3$. Also, permuting $(x,y,z)$ gives a linear action of $S_3$ on $a$ and $b$, with $x^2 \geq y^2 \geq z^2$ corresponding to $a$, $b \geq 0$. So we can assume that $a$, $b \geq 0$, and then we must show that there is a unique solution with $x \geq y \geq z \geq 0$.

The equations $x^2-y^2=a$ and $y^2 - z^2 = b$ are solved by taking $x = \sqrt{z^2+a+b}$, $y = \sqrt{z^2+b}$. The remaining equation is $t = \sqrt{z^2+a+b} \cdot \sqrt{z^2+b} \cdot z$. As $z$ increases from $0$ to $\infty$, the function $\sqrt{z^2+a+b} \cdot \sqrt{z^2+b} \cdot z$ increases monotonically from $0$ to $\infty$. So, for each $t \in [0, \infty)$, there is a unique $z$ with $t = \sqrt{z^2+a+b} \cdot \sqrt{z^2+b} \cdot z$, and then $(\sqrt{z^2+a+b},\ \sqrt{z^2+b}\ z)$ is the unique solution with $x \geq y \geq z \geq 0$ that we seek.

Before proceeding, it is convenient to make a linear change of coordinates in the above solution: Put $$c = x^2 - \tfrac{1}{2} y^2 - \tfrac{1}{2} z^2 = a + \tfrac{1}{2} b \ \text{and}\ d = \tfrac{\sqrt{3}}{2} y^2 - \tfrac{\sqrt{3}}{2} z^2 = \tfrac{\sqrt{3}}{2} b.$$ So $(c,d,t)$ is a linear change of coordinates away from $(a,b,t)$, and thus $(x,y,z) \mapsto (c,d,t)$ also has fibers equal to the orbits of $K$.

The tetrahedral group We now consider the group of symmetries of the tetrahedron, which can be written as $K \rtimes C_3$, where $C_3$ acts by cyclic permutation matrices. The reason that we worked so hard to build a new quotient by $K$ in the previous example is that the ring $\RR[a,b,t] = \RR[x^2-y^2, y^2-z^2, xyz]$ inherits an action of $C_3$. (In contrast, our general dihedral group solution gave us a different subring, $\RR[x^2-y^2, 4 x^2 y^2 - z^2, 2 xyz]$, which is not taken to itself under cyclic permutations of the variables.)

Explicitly, the action of $C_3$ fixes $t=xyz$ and acts linearly on $\RR a + \RR b$. In the linear change of coordinates $\RR a + \RR b = \RR c + \RR d$, this is the cyclic action of $C_3$ that we discussed above. So $(\text{Re}((c+id)^3),\ \text{Im}((c+id)^3),\ t)$ is a triple of polynomials as required.

The cubical case This example is like the previous one; the group is $K \rtimes S_3$, where the even permutations in $S_3$ act by permutation matrices and the odd permutations act by negations of the corresponding permutation matrix. (For example, $(x,y,z) \mapsto (-x, -z, -y)$. Note that this example preserves $c$ and negates $d$ and $t$.) As before, we get an action of $S_3$ on the ring $\RR[c,d,t]$, which is the dihedral action of $S_3$ that we discussed above. So, composing our map $(x,y,z) \mapsto (c,d,t)$ with the dihedral quotient by $S_3$, we get the desired map.

I don't know how to handle the icosahedral case. Since the group in question is $A_5$, which is simple, we can't hope to build up the solution in steps, as we did in the previous solvable cases.