A version of the spectral theorem for group actions

Question

Suppose $G$ is a sufficiently nice (maybe locally compact and abelian) group which acts on the separable Hilbert space $\mathcal{H}$ by unitary transformations. Is there a generalization of the spectral theorem to this context? Specifically, what ought the spectrum of such a group action be, and, do generalizations of spectral measures exist in this context?

This is motivated by the following line of thinking. A unitary operator $U$ naturally induces a $\mathbb{Z}$-action on $\mathcal{H}$ via $ k \cdot \psi = U^k \psi $. Then, one version of the spectral theorem produces Borel measures $\mu_\psi^U$ supported on $\mathbb{T}$ with the property that $$ \langle \psi, U^k \psi \rangle = \langle \psi, k \cdot \psi \rangle = \int_{\mathbb{T}} \! z^k \, d\mu_\psi^U(z). $$ for all $ \psi \in \mathcal{H}, k \in \mathbb{Z} $. There are a few observations that one can make. First, the spectrum of $U$ lives in the dual group $ \widehat{\mathbb{Z}} = \mathbb{T} $. Also, if $ E_k:\mathbb{T} \to \mathbb{T} $ is the map $E_k(z) = z^k$, then $ k \mapsto E_k $ is the usual Pontryagin isomorphism $ \mathbb{Z} \to \widehat{\mathbb{T}} $.

Similarly, a bounded self-adjoint operator $A$ acting on $\mathcal{H}$ induces a (continuous) unitary action of $\mathbb{R}$ on $\mathcal{H}$ via $ t \cdot \psi = e^{-itA} \psi $. Again, the spectral theorem produces measures $ \mu = \mu_\psi^A $ supported on $\mathbb{R}$ such that $$ \langle \psi, e^{-itA} \psi \rangle = \langle \psi, t \cdot \psi \rangle = \int_{\mathbb{R}} \! e^{-itx} \, d\mu_\psi^A(x), $$ and the spectrum lives in the dual group $ \widehat{\mathbb{R}} = \mathbb{R} $. This leads me to wonder the following: given a (sufficiently nice) action of a (sufficiently nice) group $G$ by unitary operators, do there exist measures $ \mu_\psi^G $ supported on $\widehat{G}$ such that $$ \langle \psi, g \cdot \psi \rangle = \int_{\widehat{G}} \! E_g(\chi) \, d\mu_\psi^G(\chi)? $$ As above, $E_g(\chi) = \chi(g)$.

This seems like a fairly reasonable question to ask, but I haven't been able to find a discussion of things like this in any of my texts on spectral theory (or any of my colleagues' texts on unitary representations).

Remark: In case this is nonstandard notation, $ \widehat{G} $ denotes the multiplicative group of continuous group homomorphisms $ G \to \mathbb{T} $, and $ \mathbb{T} = \{ z \in \mathbb{C} : |z| = 1\} $.

Answer 1

This is true for locally compact abelian $G$; you ought to be able to find it in any text on abstract harmonic analysis (a reference I have at hand is Theorem 4.44 in Folland's A Course in Abstract Harmonic Analysis).

Here's a bit of general perspective that may be helpful. Unitary representations of $G$ are the same as representations of the group C*-algebra $C^*(G)$ (essentially by definition of $C^*(G)$), and the spectrum of $C^*(G)$ can be identified with $\widehat{G}$. By the classification of commutative C*-algebras, this identifies $C^*(G)$ with $C_0(\widehat{G})$, the algebra of functions on $\widehat{G}$ vanishing at infinity. For any locally compact Hausdorff space $X$, there is a spectral theorem for representations of $C_0(X)$ whose proof is not much harder than the case of a single normal operator (which is just the case when $X$ is a compact subset of $\mathbb{C}$).

Answer 2

It sounds like you want the Arveson spectrum.