# A version of the spectral theorem for group actions

Suppose $G$ is a sufficiently nice (maybe locally compact and abelian) group which acts on the separable Hilbert space $\mathcal{H}$ by unitary transformations. Is there a generalization of the spectral theorem to this context? Specifically, what ought the spectrum of such a group action be, and, do generalizations of spectral measures exist in this context?

This is motivated by the following line of thinking. A unitary operator $U$ naturally induces a $\mathbb{Z}$-action on $\mathcal{H}$ via $k \cdot \psi = U^k \psi$. Then, one version of the spectral theorem produces Borel measures $\mu_\psi^U$ supported on $\mathbb{T}$ with the property that $$\langle \psi, U^k \psi \rangle = \langle \psi, k \cdot \psi \rangle = \int_{\mathbb{T}} \! z^k \, d\mu_\psi^U(z).$$ for all $\psi \in \mathcal{H}, k \in \mathbb{Z}$. There are a few observations that one can make. First, the spectrum of $U$ lives in the dual group $\widehat{\mathbb{Z}} = \mathbb{T}$. Also, if $E_k:\mathbb{T} \to \mathbb{T}$ is the map $E_k(z) = z^k$, then $k \mapsto E_k$ is the usual Pontryagin isomorphism $\mathbb{Z} \to \widehat{\mathbb{T}}$.

Similarly, a bounded self-adjoint operator $A$ acting on $\mathcal{H}$ induces a (continuous) unitary action of $\mathbb{R}$ on $\mathcal{H}$ via $t \cdot \psi = e^{-itA} \psi$. Again, the spectral theorem produces measures $\mu = \mu_\psi^A$ supported on $\mathbb{R}$ such that $$\langle \psi, e^{-itA} \psi \rangle = \langle \psi, t \cdot \psi \rangle = \int_{\mathbb{R}} \! e^{-itx} \, d\mu_\psi^A(x),$$ and the spectrum lives in the dual group $\widehat{\mathbb{R}} = \mathbb{R}$. This leads me to wonder the following: given a (sufficiently nice) action of a (sufficiently nice) group $G$ by unitary operators, do there exist measures $\mu_\psi^G$ supported on $\widehat{G}$ such that $$\langle \psi, g \cdot \psi \rangle = \int_{\widehat{G}} \! E_g(\chi) \, d\mu_\psi^G(\chi)?$$ As above, $E_g(\chi) = \chi(g)$.

This seems like a fairly reasonable question to ask, but I haven't been able to find a discussion of things like this in any of my texts on spectral theory (or any of my colleagues' texts on unitary representations).

Remark: In case this is nonstandard notation, $\widehat{G}$ denotes the multiplicative group of continuous group homomorphisms $G \to \mathbb{T}$, and $\mathbb{T} = \{ z \in \mathbb{C} : |z| = 1\}$.

This is true for locally compact abelian $G$; you ought to be able to find it in any text on abstract harmonic analysis (a reference I have at hand is Theorem 4.44 in Folland's A Course in Abstract Harmonic Analysis).
Here's a bit of general perspective that may be helpful. Unitary representations of $G$ are the same as representations of the group C*-algebra $C^*(G)$ (essentially by definition of $C^*(G)$), and the spectrum of $C^*(G)$ can be identified with $\widehat{G}$. By the classification of commutative C*-algebras, this identifies $C^*(G)$ with $C_0(\widehat{G})$, the algebra of functions on $\widehat{G}$ vanishing at infinity. For any locally compact Hausdorff space $X$, there is a spectral theorem for representations of $C_0(X)$ whose proof is not much harder than the case of a single normal operator (which is just the case when $X$ is a compact subset of $\mathbb{C}$).