Ring of invariants of some special type of subgroups of $GL_3(\mathbb C)$ For $\sigma \in \mathrm{GL}_n(\mathbb C)$ and $f(x_1,...,x_n)\in \mathbb C[x_1,...,x_n]$, let $f^ \sigma (x):=f(\sigma^{-1}x)$, for $x=(x_1,...,x_n)$. 
For a subgroup $G$ of $\mathrm{GL}_n(\mathbb C)$, let $\mathbb C[x_1,...,x_n]^G :=\{f\in \mathbb C[x_1,...,x_n] : f^\sigma =f ,\forall \sigma \in G\}$.
Now let $f_0,f_1,f_2 \in \mathbb C[x,y,z]$ be defined as $f_0(x,y,z)=x^3+y^3+z^3+6xyz$, $$f_1(x,y,z)=3(x^2y+y^2z+z^2x),\quad f_2(x,y,z)=3(xy^2+yz^2+zx^2).$$ 
Let $G=\{\sigma \in \mathrm{GL}_3(\mathbb C) : f_0^\sigma =f_0,\; (f_1^\sigma)^3=f_1^3,\; (f_2^\sigma)^3=(f_2)^3  \}$ .
Let $H=\{\sigma \in \mathrm{GL}_3(\mathbb C) : f_0^\sigma =f_0,\; f_1^\sigma=f_1,\; f_2^\sigma=f_2  \}$.
What are the orders of $G$ and $H$ ? Is $H$ a proper subgroup of $G$ ? How to show that every matrix in $G$ is unitary ?  Is it true that $\mathbb C[x,y,z]^G=\mathbb C[f_0,f_1^3,f_2^3]$ ? Is it true that $\mathbb C[x,y,z,]^H=\mathbb C[f_0,f_1,f_2]$ ?
 A: It is known that the linear transformation fixing $f_0$ are exactly the transformations $(x_1,x_2,x_3)\mapsto (\lambda x_{\sigma(1)}, \mu x_{\sigma(2)}, (\lambda\mu)^{-1}x_{\sigma(3)})$, where $\sigma\in\mathfrak{S}_3$ and $\lambda^3=\mu^3=1$.
These transformations also satisfy $f_i^\sigma=\gamma_if_i$ for some $\gamma_i$ such that $\gamma_i^3=1$, for $i=1,2$. Hence $G$ consists exactly of these $54$ transformations. Notice they are all unitary.
Now for such a transformation to be in $H$, you need $\lambda^2\mu=1$ and $\lambda\mu^2=1$, that is $\lambda=\mu$, so the elements of $H$ are the transformations $(x_1,x_2,x_3)\mapsto (\lambda x_{\sigma(1)}, \lambda x_{\sigma(2)}, \lambda x_{\sigma(3)})$, where $\sigma\in\mathfrak{S}_3$ and $\lambda^3=1$, which has $18$ elements, so $H$ is a proper subgroup of $G.$
(Note that it is easy to understand the group structure of $G$ and $H$, but I'm a bit lazy now).
Anyway, a polynomial fixed by $G$ or $H$ will have to be symmetric, since $G$ and $H$ contain the symmetric group on three letters, so you cannot expect $f_1$ or $f_2$ to be fixed by $H$. 
It seems that $\mathbb{C}[x_1,x_2,x_3]^H=\mathbb{C}[\sigma_1^3,\sigma_2^3,\sigma_3,\sigma_1\sigma_2]$ and $\mathbb{C}[x_1,x_2,x_3]^G=\mathbb{C}[\sigma_1^3,\sigma_2^3,\sigma_3]$, where $\sigma_1,\sigma_2,\sigma_3$ are the elementary symmetric polynomials, but I may be wrong, since I'm not an expert in invariant theory.
A: Note added on 26 Nov 2018: I have corrected my answer, which had a serious mistake.
For simplicity of notation, let $(x,y,z) = (x_1,x_2,x_3)$.  The Hessian form associated to $f_0 = {x_1}^3+{x_2}^3+{x_3}^3+6x_1x_2x_3$ is 
$$
H(f_0) = \frac{\partial^2f_0}{\partial x_i\partial x_j}\,\mathrm{d}x_i\circ\mathrm{d}x_j\,.
$$
The determinant of this Hessian form is easily computed to be
$$
\Delta = -6^3 ({x_1}^3+{x_2}^3+{x_3}^3-3x_1x_2x_3)\,
(\mathrm{d}x_1\wedge\mathrm{d}x_2\wedge\mathrm{d}x_3)^{\otimes2}.
$$
Any linear transformation of $\mathbb{C}^3$ that preserves $f_0$ must preserve $\Delta$. Since 
$$
{x_1}^3{+}{x_2}^3{+}{x_3}^3{-}3x_1x_2x_3 = (x_1{+}x_2{+}x_3)(x_1{+}sx_2{+}s^2x_3)(x_1{+}s^2x_2{+}sx_3)
$$
where $s^2+s+1=0$ (so that $s^3=1$), it follows that any linear transformation $L$ of $\mathbb{C}^3$ that preserves $f_0$ must must permute and scale the three linearly independent forms
$$
\xi_i = (x_1{+}s^ix_2{+}s^{2i}x_3)\quad i = 0,1,2.
$$
Thus, $L^*(\xi_i) = \lambda_i\,\xi_{\pi(i)}$ for $i=0,1,2$, where the $\lambda_i$ are nonzero and $\pi$ is a permutation of $\{0,1,2\}$.  Expressing $f_0$ in terms of the $\xi_i$, we find that $f_0 = \tfrac13({\xi_0}^3{+}{\xi_1}^3{+}{\xi_2}^3)$.
It now follows that $\lambda_i^3 = 1$ for $i=0,1,2$. 
Conversely, for any permuation $\pi$ of $\{0,1,2\}$ and any cube roots of unity $\lambda_i$ for $i=0,1,2$, the formula $L^*(\xi_i) = \lambda_i\,\xi_{\pi(i)}$ defines a linear transformation of $\mathbb{C}^3$ that preserves $f_0$.
Thus, the stabilizer of $f_0$ is a finite group of order $27\cdot 6 = 162$ (which is unitary for the Hermitian form $F=|\xi_1|^2{+}|\xi_2|^2{+}|\xi_3|^2 = 3(|x_1|^2{+}|x_2|^2{+}|x_3|^2)$).
Note, though, that this shows that the first sentence of GreginGre's answer is incorrect (and renders the rest of that argument moot).
Meanwhile, we find
$$
f_1 = \tfrac13({\xi_0}^3+s^2\,{\xi_1}^3+s^4\,{\xi_2}^3)
\qquad\text{and}\qquad
f_2 = \tfrac13({\xi_0}^3+s\,{\xi_1}^3+s^2\,{\xi_2}^3).
$$
It follows that $L$, as defined above, preserves $(f_i)^3$ if and only if $\pi$
is an even permutation of $\{0,1,2\}$.  Thus, $G$ has order $81$.
In order to preserve $f_1$ and $f_2$, the permutation $\pi$ must be the identity.
Thus, $H$ has order $27$.
One sees that $\mathbb{C}[x,y,z]^G$ contains $f_1f_2$, so it is not equal to $\mathbb{C}[f_0,(f_1)^3,(f_2)^3]$.
On the other hand $H$ is the group that preserves each of the ${\xi_i}^3$ (since these are linear combinations of $f_0,f_1,f_2$ and vice versa)
and $H$ preserves a monomial ${\xi_0}^{n_0}{\xi_1}^{n_1}{\xi_2}^{n_2}$ if and only if all of the $n_i$ are divisible by $3$.   Hence $\mathbb{C}[x,y,z]^H=\mathbb{C}[{\xi_0}^3,{\xi_1}^3,{\xi_2}^3]=\mathbb{C}[f_0,f_1,f_2]$.
I think that answers all your questions.
