# invariant theory for non-polynomial functions (eg Hilbert spaces)

I am looking for references regarding the study of group invariant functions that are not polynomials. In particular, I have a (nice) group $$G$$ and I am interested in what can be said about the $$G$$-invariant functions in $$L^2(X, \mu)$$ for some (nice) $$X$$ and $$\mu$$. I would be happy to understand more general spaces but $$L^2$$ seemed like a sensible start.

I am not sure if this is even well-studied as I've not been able to find anything online apart from for polynomials. Any help or pointers towards material on this topic is much appreciated. Basically, I'd just like to know what's out there, what the big theorems are and what types of questions people ask.

Edit:

This failed attempt might give some idea of the direction I am trying to go in. Let $$G$$ be a compact group acting measurably on $$X$$ and let $$\lambda$$ be the Haar measure on $$G$$. We can form the operator $$\mathcal{O}:L^2(X, \mu) \to L^2(X, \mu)$$ where $$\mathcal{O}f(x) = E_{g \sim \lambda}f(gx)$$. If $$\mathcal{O}$$ is compact, then we can apply the spectral theorem to find a finite orthonormal basis for all $$G$$-invariant functions in $$L^2(X, \mu)$$. Sadly, in most cases I care about this operator is not compact!

What I would like to know is whether there are results like this about the representation of invariant functions. It seems that invariant theory does this for polynomials, but I would like to consider more general functions.

• This is an incredibly broad question. Is there a more specific subquestion that you could pose? Some of what you ask about is usually covered as part of representation theory of groups on Hilbert spaces, but that is a vast subject. Also, when you speak of $G$-invariant functions in $L^2(X,\mu)$, are you tacitly assuming that $G$ is acting on $X$ in some measure-preserving way? Sep 14 '20 at 21:03
• It's also a bit puzzling to read someone asking "what are the big theorems" because it's not clear what you want to know about the subspace of G-invariant functions. What is an example of what you consider a "big theorem" in the area of maths that you're more familiar with? Sep 14 '20 at 21:06
• In $L^2$ the question kind of degenerates, doesn’t it? We should just get $L^2(X/G)$ with the pushforward measure, right? In other words, functions constant on the orbits. Sep 15 '20 at 0:03
• @QiaochuYuan yes, I think you're right. Sep 15 '20 at 8:21
• @YemonChoi. One thing in particular that I would be interested to know is whether there is some form of basis for this space of functions. Moreover, is it possible to take a basis for this set and extend it to a basis for the whole of L^2? Is the basis orthogonal? These are the type of things I had in mind. Also yes, I am tacitly assuming that $G$ acts measurably on $X$. In my application I would likely also take $G$ compact and $\mu$ $\sigma$-finite (eg to use fubini etc.). Sep 15 '20 at 8:21