A bit belatedly, but perhaps marginally usefully: first, yes, indeed, all you needed was that $f\in C^o_c(G)$ gave a compact operator, not even necessarily Hilbert-Schmidt or trace-class, tho' the latter does lead to further interesting things.
Second, about integrals, Gelfand and Pettis (c. 1928) effectively created a very nice notion of "integral" which works wonderfully for continuous, compactly-supported functions with values in any locally-convex, quasi-complete tvs. (These integrals are sometimes called "weak", but this is misleading in several ways.) The characterization is "weak" in the sense that $\int_X f\in V$ is uniquely determined by the fact that, for every continuous linear functional $\lambda$ on $V$, $\lambda(\int_X f)=\int_X \lambda\circ f$, where the scalar-valued integral of continuous, compactly-supported is unambiguous. One also proves that the integral is in the closure of the convex hull of the image $f(X)$, which gives a grip on "estimates". One immediate corollary is that for any continuous linear $T:V\rightarrow W$, the integral of $Tf$ is $T$ of the integral of $f$, which leads to justification of differentiation under integrals, and such. Further, giving operators on a Hilbert space the "strong operator topology" (not norm...) by $p(T)=\sup_{|x|\le 1} |Tx|$, which is what makes $G\times V\rightarrow V$ continuous, actually the operator-valued integral $\int_G f(g)\,T(g)\,dg$ makes sense, for $f\in C^o_c(G)$. E.g., see here . Various natural continuations of this, such as vector-valued holomorphic functions, were treated by Schwartz and Grothendieck. E.g., see here . Very handy on occasion.
Third, about repns "decomposing discretely, with finite multiplicities". I think for practical purposes it's not so clear what a "decomposition" would mean outside the Hilbert space context, although notions of compact or trace-class or nuclear operators have senses in larger contexts. I've heard gossip about concerted efforts at Yale in the 1950s to be able to talk about "spectral theory" in more general contexts, but it seems that it just doesn't work very well beyond Hilbert spaces, and the Fredholm alternative for special operators on Banach spaces. That such a decomposition succeeds for $L^2(compact-quotient)$ was arguably known to several people in the 1950s already: probably Gelfand et alia, but also Selberg and others, and certainly Langlands by the early 1960s. Probably those people would say that the compact quotient case was "obvious", and that the issue of serious interest was the non-compact quotient case, where one has to do some serious work to prove that the operators are still trace-class on cuspforms. Probably Gelfand-PS gave the first more-or-less proof of that, although the reader has pretty substantial responsibilities there.
The "continuous" decompositions we know for general unitary repns of type I groups are not very useful, unfortunately, in that they give no particular information. Indeed, one can execute the proof that Eisenstein series span various bits of continuous spectra... literally decomposing $L^2$ functions... without even formulating the general notion, somewhat like we can prove Fourier inversion without explaining how to view $L^2(\mathbb R)$ as the Hilbert direct integral of one-dimensional repns...
