Discrete-compact duality for nonabelian groups A standard property of Pontrjagin duality is that a locally compact Hausdorff abelian group is discrete iff its dual is compact (and vice versa). In what senses, if any, is this still true for nonabelian groups?
I can guess what this means for a compact (Hausdorff) group $G$: the category of unitary representations of $G$ should be discrete in the sense that every one-parameter family of unitary representations consists of isomorphic representations, or something like that. Is this true? Is the converse true?
I am less sure what this means for a discrete group $G$. What does it mean for the category of unitary representations to be compact? I suppose that $\text{Hom}(G, \text{U}(n))$ is a closed subspace of $\text{U}(n)^{G}$, hence compact, hence so is the appropriate quotient space of it...
 A: As it has been mentioned, the dual object to an arbitrary locally compact group is a Kac algebra, a Hopf-von Neumann algebra with some additional structures. Within the class of Kac algebras, there is a full duality theory coinciding with the Pontryagin one for Abelian groups. In particular, there is a notion of a discrete Kac algebra dual to compact ones. The definitions are too complicated to give them here. See
M. Enock and J.-M. Schwartz. Kac algebras and duality of locally compact groups, Springer, 1992.
A: I can only provide an interpretation of the discreteness for the spectrum of a compact group in terms of discrete and continouos projection valued measures. Note every measure can be uniquely decomposed in a atomic measure and a measure without atoms.
Perhaps, one should also mention that the dual of a locally compact group has also been regarded long time just as measure space. The decomposition of the group von Neumann algebra $L(G)$ of a compact group $G$ decomposes then into a direct sum of unitary irreducible representation, which are finite dimensional
$$L(G) = \bigoplus_{\pi \; irred.} M_{dim (\pi)} ( End_{G} (\pi)).$$
This decomposition is in my opinion the analogue of a noncommutative functional calculus, with the projection valued measures appearing discretely. So the spectrum of compact groups is discrete in this sense. So maybe, you're looking for the decomposition of the von Neumann group algebra of a locally compact group? For general noncompact groups, there will (necessary?) appear direct integrals. Wether these direct integrals are over compact spaces, if $G$ is discrete, I can not answer!
Factors are classified by type $1-3$. The classification for type $1$ groups, meaning that every factor appearing in the decomposition is type $1$, are easier to handle than the other types. Compact groups and abelian are of type $1$. Being type $1$ is also equivalent to certain regularity conditions on the measure space structure of the irreducible dual.
But the irreducible unitary representation of a discrete groups can be really hard to describe, e.g. a discrete group is only of type I if it contains a normal abelian
subgroup of finite index. I'll give an example: take the free group $F_n$ in $n$ generators, then the group vNa $L(F_n)$ is a factor, so in this case your "integral" is over one point. It is not known wether $L(F_n) \cong L(F_m)$ for any $n \neq m$. 
Addenum: I also have another idea for a topological picture. The Kernel-Hull topology on the primitive ideal space of $L(G)$ would be discrete for $G$ being compact.
A: Just an example, from which I learned a lot. Consider the free group $F$ on two generators. It makes a lot of sense to think of $n \mapsto hom(F,U(n))$ as some sort of dual. It comes with a natural conjugation action of $U(n)$ and natural operations of $\oplus$ and $\otimes$.
However, if one considers the bi-dual, which in this context would be the set of natural transformation $F^{xx}$ from the functor $n \mapsto hom(F,U(n))$ to $n \mapsto U(n)$, compatible with conjugation, $\oplus$ and $\otimes$, then this turns out to be too big and not discrete. First of all, $F^{xx}$ is a polish group and there is a natural homomorphism $F \to F^{xx}$. The whole construction goes under the name Chu duality and works just as in the case of Pontrjagin duality or Tannaka-Krein duality. But $F \to F^{xx}$ is not a homeomorphism. 
This follows from the fact that for fixed $n \in \mathbb N$ and a fixed neighborhood $V$ of $1_n \in U(n)$, there exists $w \in F \setminus \lbrace e\rbrace$, such that $\phi(w) \in V$, for all homomorphisms $\phi \colon F \to U(n)$. The same problem appears for every finitely generated group, which is not virtually abelian.
The problem can be cured completely if one takes the infinite-dimensional representations into account. Then, the appropriate bi-dual is equal to $F$ (and the same holds for any locally compact group). An important step in the proof of this assertion is the Gelfand-Raikov theorem.
A: No surprise, quantum groups give the right framework:
Van Daele, A., An algebraic framework for group duality. Adv. Math. 140 (1998), no. 2, 323–366. 
Summary: "A Hopf algebra is a pair $(A,\Delta)$ where $A$ is an associative algebra with identity and $\Delta$ a homomorphism from $A$ to $A\otimes A$ satisfying certain conditions. If we drop the assumption that $A$ has an identity and if we allow $\Delta$ to have values in the so-called multiplier algebra $M(A\otimes A)$, we get a natural extension of the notion of a Hopf algebra. We call this a multiplier Hopf algebra. The motivating example is the algebra of complex functions with finite support on a group with the comultiplication defined as dual to the product in the group. Also for these multiplier Hopf algebras, there is a natural notion of left and right invariance for linear functionals (called integrals in Hopf algebra theory). We show that, if such invariant functionals exist, they are unique (up to a scalar) and faithful. For a regular multiplier Hopf algebra $(A,\Delta)$ (i.e., with invertible antipode) with invariant functionals, we construct, in a canonical way, the dual $(\hat A,\hat\Delta)$. It is again a regular multiplier Hopf algebra with invariant functionals. It is also shown that the dual of $(\hat A,\hat\Delta)$ is canonically isomorphic with the original multiplier Hopf algebra $(A,\Delta)$. It is possible to generalize many aspects of abstract harmonic analysis here. One can define the Fourier transform; one can prove Plancherel's formula. Because any finite-dimensional Hopf algebra is a regular multiplier Hopf algebra and has invariant functionals, our duality theorem applies to all finite-dimensional Hopf algebras. Then it coincides with the usual duality for such Hopf algebras. However, our category of multiplier Hopf algebras also includes, in a certain way, the discrete (quantum) groups and the compact (quantum) groups. Our duality includes the duality between discrete quantum groups and compact quantum groups. In particular, it includes the duality between compact abelian groups and discrete abelian groups. One of the nice features of our theory is that we have an extension of this duality to the non-abelian case, but within one category. This is shown in the last section of our paper where we introduce the algebras of compact type and the algebras of discrete type. We prove also that these are dual to each other. We treat an example that is sufficiently general to illustrate most of the different features of our theory. It is also possible to construct the quantum double of Drinfelʹd within this category. This provides a still wider class of examples. So, we obtain many more than just the compact and discrete quantum within this setting.'' Copyright 1998 Academic Press.
A: Disclaimer: this may relate to Andreas's answer, which I don't claim to fully understand.
When asking about this discrete/compact duality, the first question should probably be "What is the appropriate analogue of the Pontryagin dual group?" Probably the underlying set should consist of irreducible unitary representations, and there is no hope of finding a group structure, but what about the topology? There is a pretty general (I'm not sure quite how general) description, due to Fell (or at least with his name attached to it), which is technically pretty horrendous and can be found at the beginning of Section 3 in the paper The Bernstein center of a $p$-adic unipotent group. I would be interested to see whether one could prove what you want in this setting.
By the way, most of the paper is devoted to proving some simpler characterizations in special cases (all totally disconnected), such as that of $p$-adic unipotent groups. There the claim is obvious.
A: There is a duality between compact groups and neutral Tannakian categories equipped with a symmetric polarization --- see Deligne and Milne, Tannakian Categories 4.27, 2.33.
