2 added rk on general notion

This is an expanded version of the comment to Marty's answer. As far as I know, this was first proved by Gelfand and Piatetskii-Shapiro. I can't look up the original journal papers, but the book

Gelʹfand, I. M.; Graev, M. I.; Pyatetskii-Shapiro, I. I. Representation theory and automorphic functions. Translated from the Russian by K. A. Hirsch W. B. Saunders Co., Philadelphia, Pa.-London-Toronto, Ont. 1969

contains the following abstract result, Lemma 2.3 in Chapter 1, p.42 of Russian edition:

If a unitary representation of $g \to T(g)$ of a locally compact group $G$ on a [Hilbert] space $H$ is such that the operator $T_\phi=\int \phi(g)T(g)dg$ is completely continuous for any finitary [i.e. compactly supported] function $\phi(g)$ then $H$ may be decomposed into a countable direct sum of irreducible unitary representations, and moreover, their multiplicities are finite.

[I've translated from Russian preserving a bit archaic language in which it was stated.] This lemma is immediately applied to prove that the representation on $L^2({\Gamma}\backslash G)$ has such a decomposition, where $G$ is a locally compact topological group and $\Gamma$ is a discrete cocompact subgroup. Later in the book, they apply it to the space of cusp forms first in the Lie group case, and then in the adelic case. These theorems require a bit more work to show that the operator is completely continuous.

They also comment on the general notion of the completely continuous representation following the proof of the lemma.

1

This is an expanded version of the comment to Marty's answer. As far as I know, this was first proved by Gelfand and Piatetskii-Shapiro. I can't look up the original journal papers, but the book

Gelʹfand, I. M.; Graev, M. I.; Pyatetskii-Shapiro, I. I. Representation theory and automorphic functions. Translated from the Russian by K. A. Hirsch W. B. Saunders Co., Philadelphia, Pa.-London-Toronto, Ont. 1969

contains the following abstract result, Lemma 2.3 in Chapter 1, p.42 of Russian edition:

If a unitary representation of $g \to T(g)$ of a locally compact group $G$ on a [Hilbert] space $H$ is such that the operator $T_\phi=\int \phi(g)T(g)dg$ is completely continuous for any finitary [i.e. compactly supported] function $\phi(g)$ then $H$ may be decomposed into a countable direct sum of irreducible unitary representations, and moreover, their multiplicities are finite.

[I've translated from Russian preserving a bit archaic language in which it was stated.] This lemma is immediately applied to prove that the representation on $L^2({\Gamma}\backslash G)$ has such a decomposition, where $G$ is a locally compact topological group and $\Gamma$ is a discrete cocompact subgroup. Later in the book, they apply it to the space of cusp forms first in the Lie group case, and then in the adelic case. These theorems require a bit more work to show that the operator is completely continuous.