3 deleted 1 characters in body

A bit belatedly, but perhaps marginally usefully: first, yes, indeed, all you needed was that $f\in C^o_c(G)$ gave a compact operator, not even necessarily Hilbert-Schmidt or trace-class, tho' the latter does lead to further interesting things.

Second, about integrals, Gelfand and Pettis (c. 1928) effectively created a very nice notion of "integral" which works wonderfully for continuous, compactly-supported functions with values in any locally-convex, quasi-complete tvs. (These integrals are sometimes called "weak", but this is misleading in several ways.) The characterization is "weak" in the sense that $\int_X f\in V$ is uniquely determined by the fact that, for every continuous linear functional $\lambda$ on $V$, $\lambda(\int_X f)=\int_X \lambda\circ f$, where the scalar-valued integral of continuous, compactly-supported is unambiguous. One also proves that the integral is in the closure of the convex hull of the image $f(X)$, which gives a grip on "estimates". One immediate corollary is that for any continuous linear $T:V\rightarrow W$, the integral of $Tf$ is $T$ of the integral of $f$, which leads to justification of differentiation under integrals, and such. Further, giving operators on a Hilbert space the "strong operator topology" (not norm...) by $p(T)=\sup_{|x|\le 1} |Tx|$, which is what makes $G\times V\rightarrow V$ continuous, actually the operator-valued integral $\int_G f(g)\,T(g)\,dg$ makes sense, for $f\in C^o_c(G)$. E.g., see here . Various natural continuations of this, such as vector-valued holomorphic functions, were treated by Schwartz and Grothendieck. E.g., see here . Very handy on occasion.

Third, about repns "decomposing discretely, with finite multiplicities". I think for practical purposes it's not so clear what a "decomposition" would mean outside the Hilbert space context, although notions of compact or trace-class or nuclear operators have senses in larger contexts. I've heard gossip about concerted efforts at Yale in the 1950s to be able to talk about "spectral theory" in more general contexts, but it seems that it just doesn't work very well beyond Hilbert spaces, and the Fredholm alternative for special operators on Banach spaces. That such a decomposition succeeds for $L^2(compact-quotient)$ was arguably known to several people in the 1950s already: probably Gelfand et alia, but also Selberg and others, and certainly Langlands by the early 1960s. Probably those people would say that the compact quotient case was "obvious", and that the issue of serious interest was the non-compact quotient case, where one has to do some serious work to prove that the operators are still trace-class on cuspforms. Probably Gelfand-PS gave the first more-or-less proof of that, although the reader has pretty substantial responsibilities there.

The "continuous" decompositions we know for general unitary repns of type I groups are not very useful, unfortunately, in that they give no particular information. Indeed, one can execute the proof that Eisenstein series span various bits of continuous spectra... literally decomposing $L^2$ functions... without even formulating the general notion, somewhat like we can prove Fourier inversion without explaining how to view $L^2(\mathbb R)$ as the Hilbert direct integral of one-dimensional repns...

Edit: after @pm's commentanswer, I realized I was not clear in what I wrote, at best:

First, yes, I was thinking only of $G$ unimodular.

Second, for many applications one wants to take trace, and trace class is a proper subset of Hilbert-Schmidt is a proper subset of "compact", altho' the composition of two Hilbert-Schmidt is trace class (perhaps by definition).

Again, the most difficult issue is proving trace class _on_cuspforms_ for integral operators attached to not-necessarily very smooth functions on a Lie group (e.g.), which Langlands did in the 1960s, I think roughly the same time as Gelfand-PS, tho' perhaps much worse documented. Adele-group-or-not is not the key point, I think. I am not aware of any systematic approach to not-co-compact $\Gamma$ in general topological groups $G$, only for reductive Lie or adele or similar.

Re: chronology, one should note that Harish-Chandra proved sharp versions of admissibility of unitaries of reductive Lie groups in the 1950s, tho' I do not know whether his results explicitly mentioned "trace class" issues.

And, yes, since the composition of two Hilbert-Schmidt ops is trace-class, the Cartier/Dixmier-Malliavin result that all smooth functions are finite sums of convolutions of smooth functions certainly crushes certain issues.

Many interesting things here! :)

2 added 1343 characters in body

Edit: after @pm's comment, I realized I was not clear in what I wrote, at best:

First, yes, I was thinking only of $G$ unimodular.

Second, for many applications one wants to take trace, and trace class is a proper subset of Hilbert-Schmidt is a proper subset of "compact", altho' the composition of two Hilbert-Schmidt is trace class (perhaps by definition).

Again, the most difficult issue is proving trace class _on_cuspforms_ for integral operators attached to not-necessarily very smooth functions on a Lie group (e.g.), which Langlands did in the 1960s, I think roughly the same time as Gelfand-PS, tho' perhaps much worse documented. Adele-group-or-not is not the key point, I think. I am not aware of any systematic approach to not-co-compact $\Gamma$ in general topological groups $G$, only for reductive Lie or adele or similar.

Re: chronology, one should note that Harish-Chandra proved sharp versions of admissibility of unitaries of reductive Lie groups in the 1950s, tho' I do not know whether his results explicitly mentioned "trace class" issues.

And, yes, since the composition of two Hilbert-Schmidt ops is trace-class, the Cartier/Dixmier-Malliavin result that all smooth functions are finite sums of convolutions of smooth functions certainly crushes certain issues.

Many interesting things here! :)

1

A bit belatedly, but perhaps marginally usefully: first, yes, indeed, all you needed was that $f\in C^o_c(G)$ gave a compact operator, not even necessarily Hilbert-Schmidt or trace-class, tho' the latter does lead to further interesting things.

Second, about integrals, Gelfand and Pettis (c. 1928) effectively created a very nice notion of "integral" which works wonderfully for continuous, compactly-supported functions with values in any locally-convex, quasi-complete tvs. (These integrals are sometimes called "weak", but this is misleading in several ways.) The characterization is "weak" in the sense that $\int_X f\in V$ is uniquely determined by the fact that, for every continuous linear functional $\lambda$ on $V$, $\lambda(\int_X f)=\int_X \lambda\circ f$, where the scalar-valued integral of continuous, compactly-supported is unambiguous. One also proves that the integral is in the closure of the convex hull of the image $f(X)$, which gives a grip on "estimates". One immediate corollary is that for any continuous linear $T:V\rightarrow W$, the integral of $Tf$ is $T$ of the integral of $f$, which leads to justification of differentiation under integrals, and such. Further, giving operators on a Hilbert space the "strong operator topology" (not norm...) by $p(T)=\sup_{|x|\le 1} |Tx|$, which is what makes $G\times V\rightarrow V$ continuous, actually the operator-valued integral $\int_G f(g)\,T(g)\,dg$ makes sense, for $f\in C^o_c(G)$. E.g., see here . Various natural continuations of this, such as vector-valued holomorphic functions, were treated by Schwartz and Grothendieck. E.g., see here . Very handy on occasion.

Third, about repns "decomposing discretely, with finite multiplicities". I think for practical purposes it's not so clear what a "decomposition" would mean outside the Hilbert space context, although notions of compact or trace-class or nuclear operators have senses in larger contexts. I've heard gossip about concerted efforts at Yale in the 1950s to be able to talk about "spectral theory" in more general contexts, but it seems that it just doesn't work very well beyond Hilbert spaces, and the Fredholm alternative for special operators on Banach spaces. That such a decomposition succeeds for $L^2(compact-quotient)$ was arguably known to several people in the 1950s already: probably Gelfand et alia, but also Selberg and others, and certainly Langlands by the early 1960s. Probably those people would say that the compact quotient case was "obvious", and that the issue of serious interest was the non-compact quotient case, where one has to do some serious work to prove that the operators are still trace-class on cuspforms. Probably Gelfand-PS gave the first more-or-less proof of that, although the reader has pretty substantial responsibilities there.

The "continuous" decompositions we know for general unitary repns of type I groups are not very useful, unfortunately, in that they give no particular information. Indeed, one can execute the proof that Eisenstein series span various bits of continuous spectra... literally decomposing $L^2$ functions... without even formulating the general notion, somewhat like we can prove Fourier inversion without explaining how to view $L^2(\mathbb R)$ as the Hilbert direct integral of one-dimensional repns...