I am looking for references regarding the study of group invariant functions that are not polynomials. In particular, I have a (nice) group $G$ and I am interested in what can be said about the $G$-invariant functions in $L^2(X, \mu)$ for some (nice) $X$ and $\mu$. I would be happy to understand more general spaces but $L^2$ seemed like a sensible start.

I am not sure if this is even well-studied as I've not been able to find anything online apart from for polynomials. Any help or pointers towards material on this topic is much appreciated. Basically, I'd just like to know what's out there, what the big theorems are and what types of questions people ask.

Edit:

This failed attempt might give some idea of the direction I am trying to go in. Let $G$ be a compact group acting measurably on $X$ and let $\lambda$ be the Haar measure on $G$. We can form the operator $$ \mathcal{O}:L^2(X, \mu) \to L^2(X, \mu) $$ where $\mathcal{O}f(x) = E_{g \sim \lambda}f(gx)$. If $\mathcal{O}$ is compact, then we can apply the spectral theorem to find a finite orthonormal basis for all $G$-invariant functions in $L^2(X, \mu)$. Sadly, in most cases I care about this operator is not compact!

What I would like to know is whether there are results like this about the representation of invariant functions. It seems that invariant theory does this for polynomials, but I would like to consider more general functions.

4more comments