MathOverflow will be down for maintenance for approximately 3 hours, starting Monday evening (06/24/2013) at approximately 9:00 PM Eastern time (UTC-4).

## a question about irreducibility of representations and Kirillov conjecture

Let $G=GL(\mathbb{R})$, $P$ be the subgroup of $G$ consisting of elements with the last row $(0,0,...,1)$. Then Kirillov conjecture states that for any irreducible unitary representation of $G$, its restriction to $P$ remains irreducible. This conjecture has been proved (not only over $\mathbb{R}$, but also over $\mathbb{C}$ and p-adic fields). Here I'm wondering if we consider irreducible smooth representations in Hilbert space(or Banach, Frechet space), does this conjecture remains true?

Another related question is generally, how to prove the irreducibility for a smooth representation besides the definition?

-
 I have a stupid question! If B is the upper triangular matrices in GL_2(R) and chi_1,chi_2 are two unitary characters of R^*, then I thought that one could make sense of the notion Ind_B^G(chi_1,chi_2) (some normalised induction to make the induced representation unitary), and that this would be an irreducible representation Pi of G. Here's the stupid bit though: if you had asked me, I would have guessed that the restriction of Pi to B (and hence to P) would have been reducible (based on an analogy with finite groups). Am I wrong? – Kevin Buzzard Mar 30 2010 at 9:41 As for your related question: I know one technique that sometimes comes up in the p-adic setting, an analogue of which might work in the real setting: if you can prove that Pi is generated as a G-rep by Pi^K, with K a compact open, and if Pi^K is irreducible as an H(G,K)-module (H the Hecke algebra: note that this is now a question about finite-dimensional representations) then in many cases this is good enough to prove that Pi is irreducible as a G-rep. – Kevin Buzzard Mar 30 2010 at 9:41

I think it's best to look at the relatively recent paper of Moshe Baruch, Annals of Math., "A Proof of Kirillov's Conjecture" -- in the introduction of his paper, he discusses the basic techniques of proof, and a bit of the history (Bernstein proved this conjecture in the p-adic case, for example).

Baruch and others (e.g. Kirillov, in the original conjecture, I think) consider unitary representations. This is necessary for the methods which they use. From the beginning, they use the "converse of Schur's lemma", i.e., if $Hom_P(V,V)$ is one-dimensional then $V$ is irreducible. This converse of Schur's lemma requires one to work in the unitary (or unitarizable) setting.

Now, to address Kevin Buzzard's point, consider $G = GL_2(F)$ for a $p$-adic field, and a unitary principal series representation $V = Ind_B^G \chi \delta^{1/2}$, where $\chi = \chi_1 \boxtimes \chi_2$ is a unitary character of the standard maximal torus, and $\delta$ is the modular character for the Borel $B$.

Restricting $V$ back down to $B$, one gets a short exact sequence of $B$-modules: $$0 \rightarrow V(BwB) \rightarrow V \rightarrow V(B) \rightarrow 0,$$ where $V(X)$ denotes a space of functions (compactly supported modulo $B$ on the left) on the ($B$-stable) locus $X$. On can check that these spaces are nonzero using the structure of the Bruhat cells, and hence the restriction of $V$ to $B$ is reducible as Kevin suggests.

But, if one considers the Hilbert space completion $\hat V$ of $V$, with respect to a natural Hermitian inner product, one finds that $\hat V$ is an irreducible unitary representation of $G$ which remains irreducible upon restriction to $B$ (and to the even smaller "mirabolic" subgroup of Kirillov's conjecture). Here it is important to note that "irreducibility" for unitary representations on Hilbert spaces refers to closed subspaces. The $B$-stable subspace $V(BwB)$ of $V$ is not closed, and its closure is all of $\hat V$ I think.

So - I think that Kirillov's conjecture is false, in the setting of smooth representations of $p$-adic groups (and most probably for smooth representations of moderate growth of real groups).

However, the techniques still apply in the smooth setting to give weaker (but still useful) results. After all, it is still useful to know that $Hom_P(V,V)$ is one-dimensional! This can be used to prove multiplicity one for certain representations, for example.

The general technique to prove $Hom_P(V,V)$ is one-dimensional involves various forms of Frobenius reciprocity and characterization of distributions. Without explaining too much (you should look at old papers of Bernstein, perhaps), and being sloppy about dualities sometimes, $$Hom_P(V,V) \cong Hom_P(End(V), C) \cong Hom_G(End(V), Ind_P^G C).$$ Some sequence of Frobenius reciprocity and linear algebra (I don't think I have it quite right above) identifies $Hom_P(V,V)$ with a space of functions or distributions: $f: G \rightarrow End(V)$, which are $(P,V)$-bi-invariant. In other words, $$f(p_1 g p_2) = \pi(p_1^{-1}) \circ f(g) \circ \pi(p_2),$$ or something close.

So in the end, one is led to classify a family of $P$-bi-quasi-invariant $End(V)$-valued distributions on $G$. This leads to two problems: one geometric, involving the $P$-double cosets in $G$. This is particularly easy for the "mirabolic" subgroup $P$. The second problem is often more difficult, analyzing distributions on each double coset, and proving most of them are zero or else have very simple properties.

Hope this clarifies a little bit... you might read more on the Gelfand-Kazhdan method (Gross has an exposition in the Bulletin) to understand this better.

-
 This was very helpful; thanks! – Emerton Mar 30 2010 at 16:19 Thanks Marty. – Kevin Buzzard Mar 30 2010 at 19:40 Thanks, Marty. I think your explanation works perfectly in p-adic setting, but in real setting, if we consider smooth Frechet (or Banach) representation, the irreducible subspaces means closed ones, so it seems still possible that $V(BwB)$ is all of $V$. Another issue in unitary case, I think, is that $B$ has Haar mesaure zero, so the function supported on it is zero in L^2. – unknown (google) Apr 1 2010 at 3:26 I don't know the real case as well as I should. But relevant aspects are described well in a paper by Casselman, Hecht, and Milicic,"Bruhat filtrations and Whittaker vectors for real groups" in the smooth setting. You can find this paper easily online. In the unitary setting, you're right that B has measure zero, so it doesn't cut out any proper subspace or quotient of the Hilbert space $L^2(G/B, \chi)$ of the unitary principal series representation. – Marty Apr 1 2010 at 5:51