MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).
show/hide this revision's text 3 typo + some remarks

For a locally compact (Hausdorff) abelian group $G$ we have following theorem (see e.g. Folland):

"For every (strongly continuous) unitary representation $(\pi,\mathcal{H_{\pi}})$ of $G$, there exists a unique regular $\mathcal{H}_{\pi}$-projection-valued measure $P$ on $\hat{G}$ such that $\pi$ decomposes as:

$\pi (g)=\int_{\hat{G}}\left\langle g,\chi\right\rangle dP\left(\chi\right)$ for every $g \in G$."

To which extent is this theorem true for nilpotent Lie groups (say, connected and simply connected)? That is, do we have a canonical decomposition of a unitary representation of such a group in terms of its irreducible unireps and a projection-valued some sort of measure on the unitary dual?

The proof of the above theorem has two major ingredients: the identification of the spectrum of $L^1 (G)$ with $\hat{G}$ when $G$ is abelian and the spectral theory of commutative Banach algebras. It is not clear to me whether any of these ingredients has a suitable analogue in the nilpotent case. Furthermore, in thie this case $\hat{G}$ is not a group or even a Hausdorff space, so I'm plus one would have to integrate an operator-valued function which assumes operators acting on different Hilbert spaces as its values. Thus I am not so sure if the standard theory of projection-valued measures can be so easily applied in this case.
show/hide this revision's text 2 minor clarification

For a locally compact (Hausdorff) abelian group $G$ we have following theorem (see e.g. Folland):

"For every (strongly continuous) unitary representation $(\pi,\mathcal{H_{\pi}})$ of $G$, there exists a unique regular $\mathcal{H}_{\pi}$-projection-valued measure $P$ on $\hat{G}$ such that $\pi$ decomposes as:

$\pi (g)=\int_{\hat{G}}\left\langle g,\chi\right\rangle dP\left(\chi\right)$ for every $g \in G$."

To which extent is this theorem true for nilpotent Lie groups (say, connected and simply connected)? That is, do we have a canonical decomposition of a unitary representation of such a group in terms of its irreducible unireps and a projection-valued measure on the unitary dualspace?

The proof of the above theorem has two major ingredients: the identification of the spectrum of $L^1 (G)$ with $\hat{G}$ when $G$ is abelian and the spectral theory of commutative Banach algebras. It is not clear to me whether any of these ingredients has a suitable analogue in the nilpotent case. Furthermore, in thie case $\hat{G}$ is not a group or even a Hausdorff space, so I'm not sure if the theory of projection-valued measures can be so easily applied in this case.
show/hide this revision's text 1

Decomposing an arbitrary unitary representation of a connected nilpotent Lie group in terms of its irreps

For a locally compact (Hausdorff) abelian group $G$ we have following theorem (see e.g. Folland):

"For every (strongly continuous) unitary representation $(\pi,\mathcal{H_{\pi}})$ of $G$, there exists a unique regular $\mathcal{H}_{\pi}$-projection-valued measure $P$ on $\hat{G}$ such that $\pi$ decomposes as:

$\pi (g)=\int_{\hat{G}}\left\langle g,\chi\right\rangle dP\left(\chi\right)$ for every $g \in G$."

To which extent is this theorem true for nilpotent Lie groups (say, connected and simply connected)? That is, do we have a canonical decomposition of a unitary representation of such a group in terms of its irreducible unireps and a projection-valued measure on the dual space?

The proof of the above theorem has two major ingredients: the identification of the spectrum of $L^1 (G)$ with $\hat{G}$ when $G$ is abelian and the spectral theory of commutative Banach algebras. It is not clear to me whether any of these ingredients has a suitable analogue in the nilpotent case. Furthermore, in thie case $\hat{G}$ is not a group or even a Hausdorff space, so I'm not sure if the theory of projection-valued measures can be so easily applied in this case.