For a locally compact (Hausdorff) abelian group $G$ we have following theorem (see e.g. Folland):
"For every (strongly continuous) unitary representation $(\pi,\mathcal{H_{\pi}})$ of $G$, there exists a unique regular $\mathcal{H}_{\pi}$-projection-valued measure $P$ on $\hat{G}$ such that $\pi$ decomposes as:
$\pi (g)=\int_{\hat{G}}\left\langle g,\chi\right\rangle dP\left(\chi\right)$ for every $g \in G$."
To which extent is this theorem true for nilpotent Lie groups (say, connected and simply connected)? That is, do we have a canonical decomposition of a unitary representation of such a group in terms of its irreducible unireps and some sort of measure on the unitary dual?
The proof of the above theorem has two major ingredients: the identification of the spectrum of $L^1 (G)$ with $\hat{G}$ when $G$ is abelian and the spectral theory of commutative Banach algebras. It is not clear to me whether any of these ingredients has a suitable analogue in the nilpotent case. Furthermore, in this case $\hat{G}$ is not a group or even a Hausdorff space, plus one would have to integrate an operator-valued function which assumes operators acting on different Hilbert spaces as its values. Thus I am not so sure if the standard theory of projection-valued measures can be so easily applied in this case.