Restriction of discrete series representations

Question

Suppose that I have a morphism of real algebraic groups $H({\mathbb R}) \to G({\mathbb R})$ with finite, abelian cokernel, and suppose that both groups admit discrete series representations. Let $\pi$ be a discrete series representation of $G({\mathbb R})$ and consider the representation of $H({\mathbb R})$ on $\pi$ by simply restricting it to $H({\mathbb R})$. My question is, which should be simple, can one explicitly describe the representations of $H({\mathbb R})$ which occur in this restriction?

I do not mind to put additional conditions, such as perhaps a regularity condition on the representation $\pi$. (But preferably no strong conditions on the groups, edit: other than ``reductive'').

EDIT: More specifically I need to know if, $\pi$ is a discrete series representation of $G({\mathbb R})$ which is sufficiently regular in some sense, does then the restriction of $\pi$ to $H(\mathbb R)$ consist of only sufficiently regular representations?

Answer 1

The answer depends on both the nature of the subgroup and the nature of the representation. For example, with $G=SL_2(\mathbb R)$, a discrete series discretely decomposes when restricted to $H=SO(2)$, but decomposes continuously for $H=GL_1(\mathbb R)$ (though this group is possibly too trivial). And if $G=SL_2(\mathbb R)\times SL_2(\mathbb R)$ and $H=SL_2(\mathbb R)$, embedded diagonally in $G$, the restriction of a holomorphic discrete series (on $G$, so this is the tensor product of two holomorphic discrete series on $H$) decomposes discretely when restricted to $H$ (as $D_n\otimes D_m\simeq \bigoplus_{k\ge 0}D_{n+m+2k}$), but the mixture of a holomorphic and an anti-holomorphic discrete series also has a continuous component when restricted (see Repka's work).

Kobayashi has published many interesting papers on this topic. See this expository paper for some necessary results and sufficient results for representations to decompose discretely. Unitary highest-weight modules (including holomorphic discrete series) are the best behaved here, but some hypotheses are still needed on the subgroup (here and Section 7 here). Basically, the tensor product (or restriction under some hypotheses) of unitary highest-weight representations decompose into unitary highest-weight representations (and if the representation is a holomorphic discrete series, the decomposition will be, too).

As an advanced example (see Section 7 in this other paper by Kobayashi), set $G=U(2,2)$, $H=Sp(1,1)$. Then $G$ has six families of discrete series, but only two decompose discretely when restricted to $H$ (ten other families of unitary representations decompose discretely).

Answer 2

I'll give some fairly general remarks - too long for a comment.

I assume $G$ is reductive and $P$ parabolic. The Mackey Restriction Induction formula yields $$ Res_{H} Ind_{P}^G \pi = \int\limits_{\gamma \in H \backslash G / P} Ind_{H \cap \gamma P \gamma^{-1}}^H \pi^\gamma \; d(\gamma)$$ Since $G/P$ is compact, the integral is over a compact space.

The Casselman submodule theorem tells you that all irreducible smooth admissible representations are obtained as submodule of $Ind_P^G \pi$. When you apply the Mackey induction restriction formula for representation for the compact subgroups, you at least obtain partial information by substraction of the $K$-types.

If you want to know whether $Ind_{H \cap \gamma P \gamma^{-1}}^H \pi^\gamma$ is irreducible, you can use Frobenius reciprocity, Mackey restriction formula and so on.

For normal subgroups, the Mackey machine describes the full unitary dual. Here you need that $H$ is reductive (or more exactly type I).

Other than that, I recall a book called "Dirac Operators in Representation Theory", where the author describes a bunch of "branching laws" (=key word). From my experience with small rank classical (mostly compact) groups, you can read it off sometimes from the Weyl character formula.

Also you might want to think about finite groups, which makes it clear that no nice things should be expected in general.