Which groups can be recovered from their unitary dual?

Question

Note: in this post, every topological group under consideration is assumed to be Hausdorff.

Given a locally compact abelian group, one can construct its dual group, i.e. its group of (unitary) characters. This turns out to be a locally compact abelian group with respect to the compact-open topology. Pontryagin duality then tells us that there is a natural isomorphism of topological groups between the original group and its double dual. Thus, if we know only the unitary characters of the group (along with their algebraic and topological structure), we can recover the original group (up to isomorphism) by taking its dual.

Similarly, given the irreducible unitary representations of a compact group, one can construct a compact group out of them which turns out to be isomorphic to the original group (the Tannaka duality theorem). Thus the irreducible unitary representations of a compact group contain enough information to recover the original group.

How much of this remains true for general locally compact groups? By the Gelfand-Raikov theorem, such groups have many irreducible unitary representations (enough to separate points). The question is: can one associate a canonical group structure and/or a topological structure to the irreducible representations of the group (or their equivalence classes) so that one recovers the original group (along with its topology) up to isomorphism?

If the answer is "no", can this be remedied by considering more general representations of the group, e.g. representations which aren't necessarily unitary?

I'm not sure if these questions have an easy answer, so I should mention that I'm mostly interested in the cases where the group is a connected nilpotent Lie group or a connected semisimple Lie group.

Answer 1

The nicest way of phrasing it is the following. Let $\mathcal H$ be the category of Hilbert spaces with unitary maps between them. For each locally compact group $G$, one can define a functor $$Rep_G : {\mathcal H} \to Top$$ with $Rep_G(H) = \hom(G,U(H))$, where the space of homomorphisms is endowed with the compact-open topology (with respect to the strong operator topology on $U(H)$). Obviously, the functor $Rep_G$ is compatible with sums and tensor products of Hilbert spaces. Note that $Rep_{\mathbb Z}(H)=U(H).$

Consider now any such functor $F: {\mathcal H} \to Top$ and set $$D(F) = Nat_{\otimes,\oplus}(F,Rep_{\mathbb Z}),$$ i.e. all natural transformations of functors which are compatible with the tensor-product and the sum. $D(F)$ is a group since $Rep_{\mathbb Z}(H)=U(H)$ is a group for each Hilbert space $H$. It is also a topological group in a natural way.

Now, there is a natural map $\iota_G : G \to D(Rep_G)$ which is given by $\iota_G(g)(\pi) = \pi(g)$, where $\pi \in hom(G,U(H))$. So just as in the case of Pontrjagin duality, there is a natural bi-dual. A bit of work (relying on results of Takesaki and Gel'fand-Raikov (which you have mentioned)) shows that $\iota_G$ is a topological isomorphism for all locally compact topological groups.

I studied the analogous question which arises if one restricts everything to the category of finite-dimensional Hilbert spaces (see here). This sometimes goes under the name Chu duality, but is not so extensively studied. Everything works for locally compact abelian groups and compact groups by Pontrjagins result and the Tannaka-Krein theorem. However, for finitely generated discrete groups, interesting things happen. First of all, it is trivial to observe that the analogous map

$$\iota_G : G \to D_{fin}(Rep_G)$$

is injective if and only $G$ is maximally almost periodic (by a result of Mal'cev iff $G$ is residually finite). Moreover, and this is more difficult, $\iota_G$ is an isomorphism if and only if $G$ is virtually abelian. In particular, for $G={\mathbb F_2}$, the map $\iota_{\mathbb F_2}$ from $\mathbb F_2$ to $D_{fin}(Rep_{\mathbb F_2})$ is not surjective. This is a bit surprising as there are no natural candidates of elements in $D_{fin}(Rep_{\mathbb F_2})$, which do not lie in the image.