Wiener algebra for nonabelian locally compact groups

By the Wiener algebra I mean the algebra of functions on the circle group having absolutely convergent Fourier series. Is there an analogue of Wiener algebra for nonabelian locally compact groups?

I am mainly interested in the compact case, but I will leave the question like this in case it may help someone else.

My thoughts: In the compact case, one possible generalization might be the space of functions $f$ such that $\sum_\pi d_\pi \| \hat{f}(\pi)\|_1 < \infty$, where the sum is taken over equivalence classes of irreducible representations, $d_\pi$ is the degree of $\pi$, $\hat{f}(\pi) = \int f(x) \pi(x)^*~dx$, and $\|\cdot\|_1$ is the Schatten $1$-norm. I would also be interested in whether this particular space has been studied.

Yes there is an analogue, and yes it is what you describe in the compact case. It is generally accepted that the "correct" generalization of the Wiener algebra to the setting of locally compact groups is the Fourier algebra, as defined by Eymard following earlier work of Stinespring for unimodular groups.

See

P. Eymard, L'algèbre de Fourier d'un groupe localement compact. Bulletin de la Société Mathématique de France, 92 (1964), 181–236 NUMDAM link

Quite a lot has been done on Fourier algebras, so it is impossible to give comprehensive references here, but at least knowing the jargon should help you to find out more depending on what you need.

(In the compact case, I think the description in terms of absolute convergence of a matrix-valued Fourier series is due to Krein, and can be found in volume 2 of Hewitt and Ross.)

Added 2014-08-06: for groups which are so-called "Type I" — a class including all compact groups, all semisimple matrix Lie groups and all nilpotent Lie groups — there is a decomposition of $L^2(G)$ as a direct integral of "multiples" of irreducibles $$L^2(G) = \int^{\oplus}_{\pi\in \widehat{G}} H_\pi \,d\nu(\pi)$$ where $d\nu$ is the so-called Plancherel measure on the unitary dual of $G$.

E.g. for $G={\bf R}$, we are viewing $L^2(\bf R)$ as $\int^{\oplus} {\bf C}_t\, dt$ where ${\bf C}_t$ is the one-dimensional representation of ${\bf R}\to {\bf T}$, $x\mapsto \exp(2\pi i tx)$.

Then — to be informal and technically a bit dodgy — the Fourier algebra $A(G)$ can be decomposed, at the level of Banach spaces, as a kind of operator-valued $L^1$-space

$$A(G) = \left\{ (T_\gamma)_{\gamma\in {\mathcal O}} \;\colon \int_{\gamma\in\mathcal{O}} \Vert T_\gamma\Vert_1 \,d\nu(\gamma) <\infty \right\}$$

for some suitable ${\mathcal O}\subseteq\widehat{G}$ which satisfies $\nu(\widehat{G}\setminus\mathcal{O})=0$. In the case $G={\bf R}$ then the Plancherel measure is just a scalar multiple of Haar measure on $\widehat{\bf R}$ and we have $A({\bf R}) \cong L^1(\widehat{\bf R})$, so that the Fourier algebra of ${\bf R}$ is the usual Wiener algebra of functions whose Fourier transforms are integrable. (Well, to be slightly more honest, it consists of the Fourier transforms of integrable functions on the dual group.)

Determining $\nu$ for particular Type I groups has been intensively studied, but at this point I should really yield the floor to those with greater knowledge on the unitary representation theory of Lie groups.

Some of the general picture can be found in Folland's A course in abstract harmonic analysis, where one or two explicit examples such as the Heisenberg group over ${\bf R}$ are worked out. A more detailed look at the general Type I case is given in Chapters 3 and 4 of Führ's book Abstract Harmonic Analysis of Continuous Wavelet Transforms, which conscientiously addresses certain technical issues about defining Fourier and inverse Fourier transforms, and how one has to be careful when your group is non-unimodular.