Decomposing an arbitrary unitary representation of a connected nilpotent Lie group in terms of its irreps For a locally compact (Hausdorff) abelian group $G$ we have following theorem (see e.g. Folland):
"For every (strongly continuous) unitary representation $(\pi,\mathcal{H_{\pi}})$ of $G$, there exists a unique regular $\mathcal{H}_{\pi}$-projection-valued measure $P$ on $\hat{G}$ such that $\pi$ decomposes as:
$\pi (g)=\int_{\hat{G}}\left\langle g,\chi\right\rangle dP\left(\chi\right)$ for every $g \in G$."
To which extent is this theorem true for nilpotent Lie groups (say, connected and simply connected)? That is, do we have a canonical decomposition of a unitary representation of such a group in terms of its irreducible unireps and some sort of measure on the unitary dual? 
The proof of the above theorem has two major ingredients: the identification of the spectrum of $L^1 (G)$ with $\hat{G}$ when $G$ is abelian and the spectral theory of commutative Banach algebras. It is not clear to me whether any of these ingredients has a suitable analogue in the nilpotent case. Furthermore, in this case $\hat{G}$ is not a group or even a Hausdorff space, plus one would have to integrate an operator-valued function which assumes operators acting on different Hilbert spaces as its values. Thus I am not so sure if the standard theory of projection-valued measures can be so easily applied in this case.
 A: First some bad news : such a decomposition only exists for groups which are said to be of Type I (some notion coming from the theory of von Neumann algebras). There are examples of topological groups which are not of Type I, and have representations which can be decomposed in two different way (even with disjoint support).
Next the good news : luckily, every connected nilpotent Lie group is of Type I. The spectral measure you are looking for indeed does exist, and in some cases (e.g., the regular representation) it is understood explicitly. See the book by Corwin and Greenleaf or the book by Pukanszky. (It is easy to find them on MathSciNet.)
A: Brad Currey has actually obtained a pretty explicit description of the Plancherel measure for general exponential solvable Lie groups which is used to decompose the representation into direct integral of irreducibles. Notice that nilpotent Lie groups are part of this larger class of exponential Lie groups. He uses the technique of jump indexes and orbit method. Here is his paper
http://mathcs.slu.edu/~currey/030401-Currey-1.pdf
Vignon S. Oussa
