When does a unitary Hilbert space rep of a reductive Lie group decompose into a direct sum of irreps with finite multiplicities?

Question

I'm giving some lectures on the trace formula. Here's something I proved in the last lecture. Let $G$ be a locally compact Hausdorff unimodular topological group (e.g. a reductive Lie group), let $\Gamma$ be a discrete subgroup with $\Gamma\backslash G$ compact, and let $H$ be $L^2(\Gamma\backslash G)$, considered as a representation of $G$ via the right regular action. Then $H$ is a Hilbert space direct sum $H=\oplus m_\pi\pi$ with $\pi$ running through irreducible unitary Hilbert space reps of $G$ and each $m_\pi<\infty$.

Although I don't think I need it for my course, it seems to me that all I really used about $H$ was that it was a unitary Hilbert space rep of $G$ and that if $f\in C_c(G)$ (continuous functions on $G$ with compact support) then (fixing a Haar measure on $G$) the induced action of $f$ on $H$ is a Hilbert-Schmidt operator. I hope that's right because I'm no expert! The point is that you can use a kernel function argument to prove that if $H=L^2(\Gamma\backslash G)$ as above then each $f$ acts via a Hilbert-Schmidt operator, but that's all you seem to need in the proof, which now goes through for any $H$ with this property.

I am minded to actually revisit the proof and remark that the arguments all (seem to) go through in this generality---not least because I'm not sure in what generality I'll need this decomposition later.

I am also minded to make the following definition:

Definition: a unitary Hilbert space rep of $G$ is Hilbert-Schmidt if each $f\in C_c(G)$ acts via a Hilbert-Schmidt operator.

Then the theorem is that Hilbert-Schmidt reps decompose into direct sums of irreps each occurring with finite multiplicity.

I am not at all sure that this is standard terminology though! Is there standard terminology to describe such reps? That's the question! Even less mathematical: is $L^2(\Gamma\backslash G)$ a representation which has some standard property $P$ (I want $P$ to be "Hilbert-Schmidt!") and the real theorem is that every rep with property $P$ decomposes into irreps each occurring with finite multiplicity?

Finally a technical point: if $H$ is a unitary rep of $G$ and $f\in C_c(G)$ then I want to define an action of $f$ on $H$ by $fv=\int_Gf(g)gv dv$. The question is how to give this integral a formal definition. Initially I had got the following definition in my mind: $fv$ is the unique element of $H$ with the property that $(fv,w)=\int_Gf(g)(gv,w) dg$ for all $w$. But now I realise there might be a "genuine" definition of $fv$ involving approximating $f$ by step functions and so on and so on. Am I right?

Answer 1

First a few background references: Read Mackey's books or articles for much more on unitary representations. Especially the Bull. AMS article "Infinite-dimensional group representations" from 1963 is a great survey to start with, and explains things like projection-valued measures, etc.. very well. You might like Garrett's notes (PDF file), about compact operators, Hilbert-Schmidt operators, and a spectral theorem.

Now, to answer the specific question: Hilbert-Schmidt is actually a bit more restrictive than you need. What you're looking for are the completely continuous representations. Here is a very general definition, from Wolf's "Harmonic analysis on commutative spaces":

Definition: A bounded Banach representation $\pi$ of a locally compact group $G$ is completely continuous if for all $f \in L^1(G)$, the operator $\pi(f)$ [defined by integration] is a compact operator. It is equivalent to assume that $\pi(f)$ is a compact operator for all $f \in C_c(G)$ by the closedness of the space of compact operators and a standard density argument.

Then Wolf follows with the basic theorem:

Theorem: Let $\pi$ be a completely continuous unitary representation of a locally compact group $G$. Then $\pi$ is a finite-multiplicity discrete direct sum of irreducible unitary representations of $G$.

Hilbert-Schmidt operators are compact, so this is a bit more general than you were suggesting. The compactness of the operators allows one to utilize the appropriate spectral theorem, and get discrete decomposability. You can also find more in "The theory of Eisenstein systems", by Osborne and Warner.

This is -- to my knowledge -- the best generality one can achieve. For more general representations, there are direct integral decompositions, with projection-valued measures, etc... that's important for understanding Eisenstein series (when the $\Gamma \backslash G$ is noncompact, for example).

Answer 2

This is an expanded version of the comment to Marty's answer. As far as I know, this was first proved by Gelfand and Piatetskii-Shapiro. I can't look up the original journal papers, but the book

Gelʹfand, I. M.; Graev, M. I.; Pyatetskii-Shapiro, I. I. Representation theory and automorphic functions. Translated from the Russian by K. A. Hirsch W. B. Saunders Co., Philadelphia, Pa.-London-Toronto, Ont. 1969

contains the following abstract result, Lemma 2.3 in Chapter 1, p.42 of Russian edition:

If a unitary representation of $g \to T(g)$ of a locally compact group $G$ on a [Hilbert] space $H$ is such that the operator $T_\phi=\int \phi(g)T(g)dg$ is completely continuous for any finitary [i.e. compactly supported] function $\phi(g)$ then $H$ may be decomposed into a countable direct sum of irreducible unitary representations, and moreover, their multiplicities are finite.

[I've translated from Russian preserving a bit archaic language in which it was stated.] This lemma is immediately applied to prove that the representation on $L^2({\Gamma}\backslash G)$ has such a decomposition, where $G$ is a locally compact topological group and $\Gamma$ is a discrete cocompact subgroup. Later in the book, they apply it to the space of cusp forms first in the Lie group case, and then in the adelic case. These theorems require a bit more work to show that the operator is completely continuous.

They also comment on the general notion of the completely continuous representation following the proof of the lemma.

Answer 3

A bit belatedly, but perhaps marginally usefully: first, yes, indeed, all you needed was that $f\in C^o_c(G)$ gave a compact operator, not even necessarily Hilbert-Schmidt or trace-class, tho' the latter does lead to further interesting things.

Second, about integrals, Gelfand and Pettis (c. 1928) effectively created a very nice notion of "integral" which works wonderfully for continuous, compactly-supported functions with values in any locally-convex, quasi-complete tvs. (These integrals are sometimes called "weak", but this is misleading in several ways.) The characterization is "weak" in the sense that $\int_X f\in V$ is uniquely determined by the fact that, for every continuous linear functional $\lambda$ on $V$, $\lambda(\int_X f)=\int_X \lambda\circ f$, where the scalar-valued integral of continuous, compactly-supported is unambiguous. One also proves that the integral is in the closure of the convex hull of the image $f(X)$, which gives a grip on "estimates". One immediate corollary is that for any continuous linear $T:V\rightarrow W$, the integral of $Tf$ is $T$ of the integral of $f$, which leads to justification of differentiation under integrals, and such. Further, giving operators on a Hilbert space the "strong operator topology" (not norm...) by $p(T)=\sup_{|x|\le 1} |Tx|$, which is what makes $G\times V\rightarrow V$ continuous, actually the operator-valued integral $\int_G f(g)\,T(g)\,dg$ makes sense, for $f\in C^o_c(G)$. E.g., see here . Various natural continuations of this, such as vector-valued holomorphic functions, were treated by Schwartz and Grothendieck. E.g., see here . Very handy on occasion.

Third, about repns "decomposing discretely, with finite multiplicities". I think for practical purposes it's not so clear what a "decomposition" would mean outside the Hilbert space context, although notions of compact or trace-class or nuclear operators have senses in larger contexts. I've heard gossip about concerted efforts at Yale in the 1950s to be able to talk about "spectral theory" in more general contexts, but it seems that it just doesn't work very well beyond Hilbert spaces, and the Fredholm alternative for special operators on Banach spaces. That such a decomposition succeeds for $L^2(compact-quotient)$ was arguably known to several people in the 1950s already: probably Gelfand et alia, but also Selberg and others, and certainly Langlands by the early 1960s. Probably those people would say that the compact quotient case was "obvious", and that the issue of serious interest was the non-compact quotient case, where one has to do some serious work to prove that the operators are still trace-class on cuspforms. Probably Gelfand-PS gave the first more-or-less proof of that, although the reader has pretty substantial responsibilities there.

The "continuous" decompositions we know for general unitary repns of type I groups are not very useful, unfortunately, in that they give no particular information. Indeed, one can execute the proof that Eisenstein series span various bits of continuous spectra... literally decomposing $L^2$ functions... without even formulating the general notion, somewhat like we can prove Fourier inversion without explaining how to view $L^2(\mathbb R)$ as the Hilbert direct integral of one-dimensional repns...

Edit: after @pm's answer, I realized I was not clear in what I wrote, at best:

First, yes, I was thinking only of $G$ unimodular.

Second, for many applications one wants to take trace, and trace class is a proper subset of Hilbert-Schmidt is a proper subset of "compact", altho' the composition of two Hilbert-Schmidt is trace class (perhaps by definition).

Again, the most difficult issue is proving trace class _on_cuspforms_ for integral operators attached to not-necessarily very smooth functions on a Lie group (e.g.), which Langlands did in the 1960s, I think roughly the same time as Gelfand-PS, tho' perhaps much worse documented. Adele-group-or-not is not the key point, I think. I am not aware of any systematic approach to not-co-compact $\Gamma$ in general topological groups $G$, only for reductive Lie or adele or similar.

Re: chronology, one should note that Harish-Chandra proved sharp versions of admissibility of unitaries of reductive Lie groups in the 1950s, tho' I do not know whether his results explicitly mentioned "trace class" issues.

And, yes, since the composition of two Hilbert-Schmidt ops is trace-class, the Cartier/Dixmier-Malliavin result that all smooth functions are finite sums of convolutions of smooth functions certainly crushes certain issues.

Many interesting things here! :)