So my question is somewhat similar to Restriction from $\mathfrak{gl}_{2n}$ to $\mathfrak{sp}_{2n}$; but I was having difficulty understanding the formula given in reference (Harris & Fulton) mentioned there.

Equation (25.37) in Harris & Fulton's "Representation Theory: A first course", says that if $m=2n$ or $m=2n+1$, $\lambda = (\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_m)$ is a $GL_m$ highest weight, and $\bar{\lambda}=(\bar{\lambda_1} \geq \cdots \geq \bar{\lambda}_n)$ is an $O_m$ highest weight, then the multiplicity of the irreducible highest-weight $O_m$ representation $V_{\bar{\lambda}}$ in the restriction of the irreducible highest-weight $GL_m$ representation $V_{\lambda}$ is equal to $\sum_{\eta} N_{\eta, \bar{\lambda}, \lambda}$; where the sum is over all partitions $\eta$ with all parts even, and $N_{\lambda_1, \lambda_2, \lambda_3}$ is the Littlewood-Richardson coefficient. I had some trouble getting this formula to work for $m=2$:

**Q1:** Since $O_m$ is disconnected, an irreducible $O_m$ representation is given by (a) if $m=2n+1$, a highest weight ${\lambda}=(\bar{\lambda_1} \geq \cdots \geq \bar{\lambda}_n)$ and a choice of $+$ or $-$ (b) if $m=2n$, a highest weight ${\lambda}=(\bar{\lambda_1} \geq \cdots \geq \bar{\lambda}_n)$ and, if $\bar{\lambda}_n=0$, a choice of $+$ or $-$. [See pg 53 of http://dspace.mit.edu/handle/1721.1/8642?show=fullfor more details.] In the above formula, the book seems to be considering $\mathfrak{o}_m$ representations instead, so how should the choices of $+$ or $-$ that arise be incorporated into the above formula?

**Q2:** Now take $m=2$. First let $\lambda=(1,1)$. If $\bar{\lambda} \neq 0$, then clearly $\sum_{\eta} N_{\eta, \bar{\lambda}, \lambda}=0$; if $\bar{\lambda}=0$, then $\sum_{\eta} N_{\eta, \bar{\lambda}, \lambda}=0$ also (since $N_{2,0,(1,1)}=0$). So this seems to be saying that the restriction of $V_{(1,1)}$ to $O(2)$ has no constituents, which is clearly false. The same problem occurs for $\lambda=(2k+1, 2k+1)$. Another example that was troubling me is $\lambda=(6,4), \bar{\lambda}=(4)$: then $N_{6, 4, (6,4)}=N_{(4,2), 4, (6,4)}=1$, so this says that the multiplicity of $V_{4}$ in the restriction of $V_{(6,4)}$ is $2$, which is also false.

`$m=2$`

is fuzzy in this context, since it involves branching to a rank one Lie subalgebra not usually included in the classification of Lie algebras of orthogonal groups. Combinatorics like this tends to require more careful interpretation for such small indices. With luck it will all make sense, but I haven't worked with these formulas recently enough to offer advice. $\endgroup$ – Jim Humphreys Feb 5 '12 at 19:49