Yes there is an analogue, and yes it is what you describe in the compact case. It is generally accepted that the "correct" generalization of the Wiener algebra to the setting of locally compact groups is the Fourier algebra, as defined by Eymard following earlier work of Stinespring for unimodular groups.
See
P. Eymard, L'algèbre de Fourier d'un groupe localement compact. Bulletin de la Société Mathématique de France, 92 (1964), 181–236
NUMDAM link
Quite a lot has been done on Fourier algebras, so it is impossible to give comprehensive references here, but at least knowing the jargon should help you to find out more depending on what you need.
(In the compact case, I think the description in terms of absolute convergence of a matrix-valued Fourier series is due to Krein, and can be found in volume 2 of Hewitt and Ross.)
Added 2014-08-06: for groups which are so-called "Type I" — a class including all compact groups, all semisimple matrix Lie groups and all nilpotent Lie groups — there is a decomposition of $L^2(G)$ as a direct integral of "multiples" of irreducibles
$$ L^2(G) = \int^{\oplus}_{\pi\in \widehat{G}} H_\pi \,d\nu(\pi) $$
where $d\nu$ is the so-called Plancherel measure on the unitary dual of $G$.
E.g. for $G={\bf R}$, we are viewing $L^2(\bf R)$ as $\int^{\oplus} {\bf C}_t\, dt$ where ${\bf C}_t$ is the one-dimensional representation of ${\bf R}\to {\bf T}$, $x\mapsto \exp(2\pi i tx)$.
Then — to be informal and technically a bit dodgy — the Fourier algebra $A(G)$ can be decomposed, at the level of Banach spaces, as a kind of operator-valued $L^1$-space
$$ A(G) = \left\{ (T_\gamma)_{\gamma\in {\mathcal O}} \;\colon \int_{\gamma\in\mathcal{O}} \Vert T_\gamma\Vert_1 \,d\nu(\gamma) <\infty \right\} $$
for some suitable ${\mathcal O}\subseteq\widehat{G}$ which satisfies $\nu(\widehat{G}\setminus\mathcal{O})=0$. In the case $G={\bf R}$ then the Plancherel measure is just a scalar multiple of Haar measure on $\widehat{\bf R}$ and we have $A({\bf R}) \cong L^1(\widehat{\bf R})$, so that the Fourier algebra of ${\bf R}$ is the usual Wiener algebra of functions whose Fourier transforms are integrable. (Well, to be slightly more honest, it consists of the Fourier transforms of integrable functions on the dual group.)
Determining $\nu$ for particular Type I groups has been intensively studied, but at this point I should really yield the floor to those with greater knowledge on the unitary representation theory of Lie groups.
Some of the general picture can be found in Folland's A course in abstract harmonic analysis, where one or two explicit examples such as the Heisenberg group over ${\bf R}$ are worked out. A more detailed look at the general Type I case is given in Chapters 3 and 4 of Führ's book
Abstract Harmonic Analysis of Continuous Wavelet Transforms, which conscientiously addresses certain technical issues about defining Fourier and inverse Fourier transforms, and how one has to be careful when your group is non-unimodular.
