# Restriction of discrete series representations

Suppose that I have a morphism of real algebraic groups $H({\mathbb R}) \to G({\mathbb R})$ with finite, abelian cokernel, and suppose that both groups admit discrete series representations. Let $\pi$ be a discrete series representation of $G({\mathbb R})$ and consider the representation of $H({\mathbb R})$ on $\pi$ by simply restricting it to $H({\mathbb R})$. My question is, which should be simple, can one explicitly describe the representations of $H({\mathbb R})$ which occur in this restriction?

I do not mind to put additional conditions, such as perhaps a regularity condition on the representation $\pi$. (But preferably no strong conditions on the groups, edit: other than reductive'').

EDIT: More specifically I need to know if, $\pi$ is a discrete series representation of $G({\mathbb R})$ which is sufficiently regular in some sense, does then the restriction of $\pi$ to $H(\mathbb R)$ consist of only sufficiently regular representations?

-
You should be more precise in what direction sufficiently regular goes. E.g., growth conditions on matrix coefficients? –  plusepsilon.de Jul 18 '12 at 18:03
It is a condition on the infinitesimal character –  Arno Kret Jul 22 '12 at 8:30
Toshiyuki Kobayashi as well as Michel Duflo have worked on branching problems for discrete series representations of real semisimple Lie groups, please, check on mathscinet for the precise references. –  user36546 Jul 4 '13 at 22:30

The answer depends on both the nature of the subgroup and the nature of the representation. For example, with $G=SL_2(\mathbb R)$, a discrete series discretely decomposes when restricted to $H=SO(2)$, but decomposes continuously for $H=GL_1(\mathbb R)$ (though this group is possibly too trivial). And if $G=SL_2(\mathbb R)\times SL_2(\mathbb R)$ and $H=SL_2(\mathbb R)$, embedded diagonally in $G$, the restriction of a holomorphic discrete series (on $G$, so this is the tensor product of two holomorphic discrete series on $H$) decomposes discretely when restricted to $H$ (as $D_n\otimes D_m\simeq \bigoplus_{k\ge 0}D_{n+m+2k}$), but the mixture of a holomorphic and an anti-holomorphic discrete series also has a continuous component when restricted (see Repka's work).

Kobayashi has published many interesting papers on this topic. See this expository paper for some necessary results and sufficient results for representations to decompose discretely. Unitary highest-weight modules (including holomorphic discrete series) are the best behaved here, but some hypotheses are still needed on the subgroup (here and Section 7 here). Basically, the tensor product (or restriction under some hypotheses) of unitary highest-weight representations decompose into unitary highest-weight representations (and if the representation is a holomorphic discrete series, the decomposition will be, too).

As an advanced example (see Section 7 in this other paper by Kobayashi), set $G=U(2,2)$, $H=Sp(1,1)$. Then $G$ has six families of discrete series, but only two decompose discretely when restricted to $H$ (ten other families of unitary representations decompose discretely).

-
Thanks for the references! I work under the (very restrictive) condition that H is with finite and abelian cokernel in G. –  Arno Kret Jul 29 '12 at 12:47
Oh! That was very silly of me: I read the condition as "kernel" rather than "cokernel" (which seemed an odd condition, but I wasn't going to judge). I'm not familiar with the implications of this condition, so I can't be sure, but I think the answer should be "yes". –  B R Jul 29 '12 at 16:28

I'll give some fairly general remarks - too long for a comment.

I assume $G$ is reductive and $P$ parabolic. The Mackey Restriction Induction formula yields $$Res_{H} Ind_{P}^G \pi = \int\limits_{\gamma \in H \backslash G / P} Ind_{H \cap \gamma P \gamma^{-1}}^H \pi^\gamma \; d(\gamma)$$ Since $G/P$ is compact, the integral is over a compact space.

The Casselman submodule theorem tells you that all irreducible smooth admissible representations are obtained as submodule of $Ind_P^G \pi$. When you apply the Mackey induction restriction formula for representation for the compact subgroups, you at least obtain partial information by substraction of the $K$-types.

If you want to know whether $Ind_{H \cap \gamma P \gamma^{-1}}^H \pi^\gamma$ is irreducible, you can use Frobenius reciprocity, Mackey restriction formula and so on.

For normal subgroups, the Mackey machine describes the full unitary dual. Here you need that $H$ is reductive (or more exactly type I).

Other than that, I recall a book called "Dirac Operators in Representation Theory", where the author describes a bunch of "branching laws" (=key word). From my experience with small rank classical (mostly compact) groups, you can read it off sometimes from the Weyl character formula.

Also you might want to think about finite groups, which makes it clear that no nice things should be expected in general.

-
Thank you. I forgot to put in the question that I am mostly interested in reductive groups. –  Arno Kret Jul 18 '12 at 16:59
Also there a certain Gelfand pairs, i.e. if $(H,G)$ is such a pair, then it decomposes with multiplicity one. GL(n+1) and GL(n) over a local field works for example. –  plusepsilon.de Jul 18 '12 at 17:08
Thank you, I have made the question more precise. See the edit. Maybe you know a response to that, more specific, question? –  Arno Kret Jul 18 '12 at 17:48