Let G be a noncompact Lie group, such as SL(n,R), GL(n,C).
How does the regular representation $L^2(G)$ decompose? Is there an analogue of PeterWeyl theorem?
Let G be a noncompact Lie group, such as SL(n,R), GL(n,C). How does the regular representation $L^2(G)$ decompose? Is there an analogue of PeterWeyl theorem? 


Consider the simplest case! By the Fourier transform, $L^2(\mathbb R,+)$ is isomorphic to $L^2 (\mathbb R)$ with the group action $x \cdot f( \xi) = f(\xi) e^{2 \pi i x \xi}$ A simple example of a subrepresentation consists of all functions that are zero outside the interval $[a,b]$. But these representations are always decomposable  e.g. into $[a,a+b/2] \cup [a+b/2,b]$. Indeed there are no indecomposible subrepresentations, because the functions involved would have to be supported at a point, but there are no $L^2$ functions supported at a given point. Thus we already get quite different behavior. Though there are many irreducible represntations, we cannot find them in the regular representation. Instead, the regular representation can be decompossed into smaller representations, that get closer and closer to these irreducible representations in some sense, but are never close enough that they themselves are indecomposable. Thus, in addition to a PeterWeyl "discrete spectrum" phenomenon, there is also this sort of "continuous spectrum". 


Adding some further detail to Will Sawin's answer, in some examples that are clearer than the general case: For semisimple real Lie groups $G$ obtained by "restricting scalars" of ("classical") complex semisimple Lie groups, GelfandNaimark in 1947 (I think...?) gave a Plancherel theorem for $L^2(G)$, involving integrals of (the simplest) irreducible unitary repns, namely, the unitary principal series. The "Plancherel measure" was/is a uniform sort of realline measure on the spectral parameters. They conjectured that these were all the irreducible unitaries of such groups, but it was observed later that there were "unitary degenerate principal series" that occurred as proper subrepns of nonunitary (and not unitarizable) principal series. After work of Wigner and Bargmann in the late 1940s on some small examples of semisimple groups not obtained by restriction of scalars on complex groups, in the early 1950s HarishChandra started considering the more general semisimple real case. Although the "holomorphic discrete series" repns of classical groups "of hermitian type" had been implicit in Siegel's and other's early work on modular forms in several complex variables since the late 1930s, it was only in the 1950s that HarishChandra made explicit the point that, for such (noncompact) groups there are nontrivial "discrete series" repns, meaning, literally ("discretely") appearing in the decomposition of $L^2(G)$, rather than appearing "continuously". It took HarishChandra many years to treat the other discrete series repns of semisimple real Lie groups, and, even then, he reasoned indirectly by constructing their characters rather than having models of the repns. In fact, constructions of models was an ongoing process in the 1960s and 1970s, with W. Schmid, R. Langlands, G. Zuckerman, and others adding new ideas of various sorts. The historical notes in A. Knapp's Princeton Press volume on repn theory of semisimple real Lie groups include more bibliographic pointers. 


The answers by Will and Paul, along with various comments, are probably enough to convince anyone that the question asked is exceptionally broad (too broad for a selfcontained answer) and involves a great deal of wellorganized theory developed in recent generations. The "group algebra" $L^2(G)$ is especially well studied for semisimple Lie groups, where you could spend considerable time studying the collected works of HarishChandra (reviewed nicely by V.S. Varadarajan in Bull. London Math. Soc. 17 (1985)). For some useful expository articles and references to related work, see the volume: The mathematical legacy of HarishChandra. A celebration of representation theory and harmonic analysis. Proceedings of the AMS Special Session on Representation Theory and Noncommutative Harmonic Analysis, held in memory of HarishChandra on the occasion of the 75th anniversary of his birth, in Baltimore, MD, January 9–10, 1998. Edited by Robert S. Doran and V. S. Varadarajan. Proceedings of Symposia in Pure Mathematics, 68. American Mathematical Society, Providence, RI, 2000 

