# Restriction of representation from GL(n) to O(n)

So my question is somewhat similar to Restriction from $\mathfrak{gl}_{2n}$ to $\mathfrak{sp}_{2n}$; but I was having difficulty understanding the formula given in reference (Harris & Fulton) mentioned there.

Equation (25.37) in Harris & Fulton's "Representation Theory: A first course", says that if $m=2n$ or $m=2n+1$, $\lambda = (\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_m)$ is a $GL_m$ highest weight, and $\bar{\lambda}=(\bar{\lambda_1} \geq \cdots \geq \bar{\lambda}_n)$ is an $O_m$ highest weight, then the multiplicity of the irreducible highest-weight $O_m$ representation $V_{\bar{\lambda}}$ in the restriction of the irreducible highest-weight $GL_m$ representation $V_{\lambda}$ is equal to $\sum_{\eta} N_{\eta, \bar{\lambda}, \lambda}$; where the sum is over all partitions $\eta$ with all parts even, and $N_{\lambda_1, \lambda_2, \lambda_3}$ is the Littlewood-Richardson coefficient. I had some trouble getting this formula to work for $m=2$:

Q1: Since $O_m$ is disconnected, an irreducible $O_m$ representation is given by (a) if $m=2n+1$, a highest weight ${\lambda}=(\bar{\lambda_1} \geq \cdots \geq \bar{\lambda}_n)$ and a choice of $+$ or $-$ (b) if $m=2n$, a highest weight ${\lambda}=(\bar{\lambda_1} \geq \cdots \geq \bar{\lambda}_n)$ and, if $\bar{\lambda}_n=0$, a choice of $+$ or $-$. [See pg 53 of http://dspace.mit.edu/handle/1721.1/8642?show=fullfor more details.] In the above formula, the book seems to be considering $\mathfrak{o}_m$ representations instead, so how should the choices of $+$ or $-$ that arise be incorporated into the above formula?

Q2: Now take $m=2$. First let $\lambda=(1,1)$. If $\bar{\lambda} \neq 0$, then clearly $\sum_{\eta} N_{\eta, \bar{\lambda}, \lambda}=0$; if $\bar{\lambda}=0$, then $\sum_{\eta} N_{\eta, \bar{\lambda}, \lambda}=0$ also (since $N_{2,0,(1,1)}=0$). So this seems to be saying that the restriction of $V_{(1,1)}$ to $O(2)$ has no constituents, which is clearly false. The same problem occurs for $\lambda=(2k+1, 2k+1)$. Another example that was troubling me is $\lambda=(6,4), \bar{\lambda}=(4)$: then $N_{6, 4, (6,4)}=N_{(4,2), 4, (6,4)}=1$, so this says that the multiplicity of $V_{4}$ in the restriction of $V_{(6,4)}$ is $2$, which is also false.

• Your case $m=2$ is fuzzy in this context, since it involves branching to a rank one Lie subalgebra not usually included in the classification of Lie algebras of orthogonal groups. Combinatorics like this tends to require more careful interpretation for such small indices. With luck it will all make sense, but I haven't worked with these formulas recently enough to offer advice. – Jim Humphreys Feb 5 '12 at 19:49
• Regarding Q2 - isn't there some rank restriction? The group $O(2)$ is not semisimple so that may cause some problems. – Vít Tuček Feb 5 '12 at 19:53
• Regarding Q1 -- I suggest you read (the relevant parts of) the book by Goodman and Wallach. See e.g. appendix B on math.rutgers.edu/~goodman/repbook.html Then you will have some feeling how these branchings behave. – Vít Tuček Feb 5 '12 at 19:57
• Thanks! So maybe I should exclude the case n=2. But then the same problem seems to occur for the $GL(n)$ representation with highest weight $(1,1,\cdots,1)$ (or $(2k+1,2k+1, \cdots ,2k+1)$), which still confuses me. rObOt, I couldn't find anything in appendix B, it seems to be about linear algebra. Do you mean perhaps appendix F (and Chapter 10)? Would you know which pages addresses this question? – Vinoth Feb 5 '12 at 23:29
• Yes, I am sorry, I meant appendix F. This appendix tells you how to decompose the $k$-th tensor product of the defining representation of $GL(n,\mathbb{C})$ with respect to either $O(n,\mathbb{C})$ or $Sp(n,\mathbb{C})$. You should also look at chapter 10. The main point is that the decomposition of the space of harmonic tensors (i.e. all traces are zero) for $GL(n)$ is the same as for $O(n)$. General tensors decompose into partially harmonic tensors (i.e. only some traces are zero) which are isomorphic to harmonic tensors of lower valence). There you can see the LR coefficients to appear. – Vít Tuček Feb 6 '12 at 19:24

For the exterior powers (highest weight $(1,1,\dots,1)$), the restriction to the orthogonal group is still irreducible -- I think you are forgetting to take $\overline{\lambda} = 0$.
There are also issues with stable ranges in this formula (you have to assume that $\lambda$ doesn't have more than $n$ parts). If you want the formulas to work in all cases, then you have to use modification rules for transforming partitions of length more than $n$ into those with length at most $n$ (sometimes you have to introduce signs to cancel terms). Details of this modification rule can be found in several sources. One such that I like is Koike--Terada's paper: http://www.sciencedirect.com/science/article/pii/0021869387900998 (Section 2.4)