There is a whole theory of finding the invariant polynomials for matrix groups $\Gamma$ acting on the polynomial ring $\mathbb{C}[x_1,\ldots,x_n]$. I would be interested in finding invariant polynomials for a different action of a matrix group on this polynomial ring. Let me give an example.

Period polynomials (of modular forms) are homogeneous elements in $\mathbb{C}[x,y]$ satisfying $$ f(x,y) + f(-y,x) = f(x,y) + f(x-y,x) + f(-y,x-y) = 0. $$ Letting $S=\left(\begin{smallmatrix} 0 & -1 \\ 1 & 0 \end{smallmatrix}\right)$ and $U=\left(\begin{smallmatrix} 1 & -1 \\ 1 & 0 \end{smallmatrix}\right)$, period polynomials can be interpreted as invariant polynomials of a non-standard action of $\langle S,U+U^2 \rangle \leq \mathbb{Z}[\mathrm{SL}_2(\mathbb{Z})]$ by $$ (Sf)(x,y) = -f(-y,x) \qquad \text{and} \qquad ((U+U^2)f)(x,y) = -f(x-y,x)-f(-y-x+y). $$

My question is: can one, without using the theory of modular forms, describe the invariant polynomials, i.e., the period polynomials?

I would also be interested in the answer to the more general question: given an action by a subgroup of the group algebra of matrices, how does one determine the invariant polynomials? To provide another example, experiments suggest there are certain polynomials satisfying $$f(x,y,z) = f(x+y,x,z)+f(x+y,y,z),$$ but none of these polynomials is also symmetric. How can one prove such a statement?