# Restriction of irreducible unitary representation to normal subgroup of finite index

Let $G$ be a Lie group (or more generally a locally compact group), let $N$ be a closed and normal subgroup of $G$ of finite index. Let $H$ be an infinite dimensional complex Hilbert space, and let $\pi$ be an irreducible unitary representation of $G$ on $H$.

Now consider the restriction $\pi|_N$ of $\pi$ to $N$. Is it true that $\pi|_N$ is the finite sum of irreducible unitary representations? Perhaps someone could provide an indication of the proof or a reference?

Or how about a more basic question: Must $(\pi|_N, H)$ have a (discretely occuring) irreducible subrepresentation? Or even just an irreducible quotient (as would exist for a finitely generated algebraic representation)?

• @studiosus - Yes thanks I have looked at that. That question was purely algebraic, without the assumption of unitarity. The proof starts out with establishing an irreducible quotient. I'd be interested if someone thought that same proof could apply in the unitary case. I am not sure if the irreducible quotient argument works or not. – grad student Jul 11 '16 at 22:30
• Very nice well written question! +1 – user64742 Jul 16 '16 at 2:37

I like this question!

Restricted to the finite index subgroup $N$, the representation $\pi$ splits into a direct sum of irreducible representations.

I could not see an easy proof of this, but the proof goes along the following lines.

Suppose $I$ is a totally ordered indexing set and for each $i\in I$, $W_i$ is an $N$ invariant non-zero (closed) subspace of $H$ such that $i<j$ implies $W_i\supset W_j$. Then I claim that the intersection $\cap _{i\in I}W_i$ is a closed non-zero $N$ invariant subspace. Let $P_i$ be the projection map from $H$ onto $W_i$. This map is $N$-equivariant. Consider the finite sum $p_i= \sum _{g\in G/N} gP_ig^{-1}$. Being a $G$ invariant self adjoint bounded operator, by Schur's lemma, $p_i$ is a scalar $c_iI$ for some $c_i\geq 0$. The scalar $c_i\geq 1$: if $v\in W_i$ has norm one, then $c_i=(p_iv,v)\geq (P_iv,v)=1$.

On the other hand, the $W_i$ form a decreasing family. Suppose $w\in H$. The net of numbers $(P_iw,w)$ decreases to $(Pw,w)$ where $P$ is the projection to the intersection $\cap W_i$. (This last statement needs a proof, which is tedious but routine. One has to replace the family $(P_iw,w)$ by a countable subfamily whose lim inf is the lower limit of the family, and then argue that the lower limit is $(Pw,w)$).

From the last two paragraphs, for any $w\in H$ of norm one, we have $c_i=(c_iw,w)=\sum _{g\in G/N}(gP_ig^{-1}w,w)$ decreases to $\sum _{g\in G/N}(gPg^{-1}w,w)$. Since $c_i\geq 1$, it follows that the projection $P$ is non-zero. Hence the intersection $\cap _{i\in I}W_i$ is non-zero.

By the Hausdorff maximality principle, there exists a minimal $N$ invariant non-zero closed subspace $W$ in $H$. This must necessarily be irreducible. The finite sum $\sum _{g\in G/N}gW$ being non-zero and $G$ invariant, is dense in $H$. Each $gW$ is also $N$-irreducible since $N$ is normal. It follows that $H$ is a direct sum of possibly a smaller collection of $gW$.

• Following your argument, for the existence of the nonzero irreducible subspace $W$, how critical is the finiteness of the index of $N$ in $G$? Could $N$ be replaced by a closed subgroup and the finite sum be replaced by an integral over the quotient space? Where does the contradiction arise? – grad student Jul 21 '16 at 0:42
• @gradstudent, the quotient space need not support an invariant measure (if the modular characters don't match), nor the integral with respect to such a measure (if there is one) converge. – LSpice Jul 21 '16 at 1:32
• @gradstudent, LSpice is right. What is more, the conjugation action on bounded operators (under the operator norm) need not be continuous, so taking integrals is problematic. Finally, even if $W$ is an $N$ invariant subspace, the "sum of $gW$" need not make sense. – Venkataramana Jul 21 '16 at 2:10
• And as a corollary, every irreducible unitary of $G$ imbeds in a repn induced from an irreducible unitary of $N$, by a (true) version of Frobenius Reciprocity, apparently!?! – paul garrett Jul 21 '16 at 13:36
• The motivation for this question was to verify the following statement by Borel in his IAS/Park City article on automorphic forms on reductive groups: "We have referred to statements in which $G$ is assumed to be connected. However, our $G$ has at most finitely many connected components. It is standard and elementary that the restriction to a normal subgroup of finite index of an irreducible representation is a finite sum of irreducible representations, so the extension to our case is immediate." However, I never found a proof in the literature so I appreciate Venkataramana's answer. – grad student Jul 21 '16 at 15:42