Suppose $G$ is a semisimple group, and $V_{\lambda}$ is an irreducible finitedimensional representation of highest weight $\lambda$. Suppose $H \subset G$ is a semisimple subgroup. What is the multiplicity of the trivial representation in $V_{\lambda}_{H}$? Is there a simple way to read this off from $\lambda$ and the Dynkin diagrams of $G$ and $H$?

$\begingroup$ I would, if I were you, give myself a little extra data. Recall that there is no good functor from groups to Dynkin diagrams  you have to make choices to associate some random semisimple group with the particular group defined by a diagram via ChevalleySerre. OTOH, if $H \subseteq G$ is a semisimple subgroup of a semisimple group, I think you can find a Cartan in $H$ and a Cartan in $G$ so that the Cartans inject, and a simple system for $H$ that is a subsimple system for $G$. I could be wrong. $\endgroup$– Theo JohnsonFreydMar 20, 2010 at 3:11

$\begingroup$ (Continued) So the point is: I think you can assume that the diagram for $H$ is a subdiagram of the diagram for $G$, and I think you should do this, because otherwise I can't image that you'd be able to write down any useful formulas. Maybe you were assuming this already, but if so you should make it explicit, and you should check whether I'm right that any injection of semisimples is conjugate to a sub Dynkin diagram. $\endgroup$– Theo JohnsonFreydMar 20, 2010 at 3:13
4 Answers
The short answer is: no. In theory, there are explicit formulae for branchings based on the Weyl character formula, but no reasonable person would call these simple.
There are interesting results which give you some information about branching multiplicities, but all involve real work. For instance, if G and H are compact, you can obtain asymptotic information about Hinvariants in the representations $V_{n\lambda}$ as a function of $n$: it's a polynomial, whose leading order is the dimension of $\mathcal{O}_{\lambda}//H $, the symplectic reduction of the coadjoint orbit through $\lambda$ by $H$, and whose leading coefficient is the symplectic volume of this manifold.
If you prefer algebraic geometry, this polynomial is the Hilbert polynomial of the corresponding GIT quotient.
If $H$ is a root subalgebra, then life is a bit easier, and you can use combinatorial methods like crystals, but this is still not easy.

$\begingroup$ It seems to be understood here that everything is done in characteristic 0 (in characteristic p it all gets much worse). To underscore Ben's point, you could start with the special linear group
$G$
and embed into it an arbitrary connected semisimple group$H$
. Then almost nothing can be said about restricting representations. Even if$H$
is generated by pairs of one dimensional root subgroups relative to a given maximal torus of$G$
, branching rules are usually quite intricate to work out. For invariants, maybe the papers of Roger Howe et al. would help? $\endgroup$ Mar 19, 2010 at 18:20 
$\begingroup$ Jim He did say "Lie group" in the title, so without other context, I think I was pretty safe in assuming characteristic 0. $\endgroup$– Ben Webster ♦Mar 19, 2010 at 20:25

$\begingroup$ Yes, I did lose the title after reading the question. This can be formulated in a number of settings: complex semisimple Lie groups or Lie algebras or compact groups or algebraic groups in any characteristic. Even formulated narrowly, the question is not easy to answer definitively. If it were, invariant theory might be less challenging. $\endgroup$ Mar 19, 2010 at 22:29

Consider the case that $H$ is a maximal torus of $G'$, and your $G = G' \times H$. (Well, you said $G,H$ semisimple, but I'm going to pretend you meant reductive, because really you should have.) Then your question is answered by the Kostant multiplicity formula.
If you're willing to take that formula as "simple", then yes, the general case is not much harder. Let $T_H$ be a maximal torus for $H$, and $T_G$ a maximal torus that contains it. Use the Kostant multiplicity formula to go from $G$ to $T_G$; now we have a function on the weight lattice $T_G^*$ of $T_G$. Push it forward under the restriction map $T_G^* \to T_H^*$. Difference that in the directions of all the positive roots of $H$. Look at the value at $0$.
If you want a positive formula (like I do!) then none is known. What you're asking for includes the case that $G = H\times H\times H$, and then the question becomes one about computing tensor product decompositions. That subcase does have positive formulae, e.g. counting Littelmann paths, but noone has extended it even to the case of branching from $H \times K$ to $H$ where $K$ is a symmetric subgroup of $H$.
I would like to add to the answers by Ben and Allen. First if we extend the question to include all multiplicities and not just the multiplicity of the trivial representation then there are a number of special cases that are of interest:
1.Take $H$ to be the trivial group then the question asks for the dimension of a representation.
2.Take $H$ to be a maximal torus then we are asking for the character of a representation.
3. Take $G=H\times H$ and $H$ the diagonal subgroup. Then we are asking for tensor product multiplicities.
4. For $V$ a representation of $K$. Take $G=SL(V)$ and $H=K$. Then we are calculating plethysms.
A paper that discusses this which gives a formula for branching rules is:
MR1120029 (92f:22022) Cohen, Arjeh M. ; Ruitenburg, G. C. M. Generating functions and Lie groups. Computational aspects of Lie group representations and related topics (Amsterdam, 1990), 1928, CWI Tract, 84, Math. Centrum, Centrum Wisk. Inform., Amsterdam, 1991.
As I understand it both Ben and Allen agree that this is not a simple way of finding branching rules. The reason is that this involves a sum over the Weyl group.
If you take the special cases above then historically the first solutions to these problems were given by formulae involving a sum over the Weyl group. For some of these special cases there are solutions which don't involve cancelling terms. For example, LiE
calculates these without summing over the Weyl group. The LiE home page is
http://wwwmath.univpoitiers.fr/~maavl/LiE/index.html
and the LiE manual does describe how these special cases are implemented.
However LiE treats each of these special cases separately. I think it is an interesting question whether there is an algorithm for finding branching rules which could be implemented in LiE and which does not involve a sum over the Weyl group.
I want to refer you to the following article by B. Cahen, treating the case of compact Lie groups, where an approach based on the realization of the irreducible representations of G according to the BorelWeil theorem as reproducing kernel Hilbert spaces within the framework of Berezin quantization is adopted. The article describes a multiplicity formula (equation  3.3) of an irreducible representation of a compact group G in the decomposition of a reproducing kernel Hilbert space on which G acts unitarily whose elements are functions on a manifold M.
When applied to the case in the question, the reproducing kernel Hilbert space can be taken as the space of holomorphic sectionsof the line bundle associated with lambda (according to the BorelWeil theorem) on the flag manifold G/T. The multiplicity formula contains a double integral, the first is over the maximal torus of H which acts as a projector onto the required representation space and the second is over the flag manifold G/T which acts as a multiplicity counter.
In this method no explicit sum on the Weyl group is needed, but it is replaced by the task of the double integration. The integration on the maximal torus is quite easy. The integration on the flag manifold looks like a formidable task but at least for the case of weight multiplicity computation it can be replaced by a computation of the rank of a Hermitian matrix (proposition 3.6). In the article some low dimensional examples are included.
This method does not give nice closed formulas like the combinatorial methods, but may be it has a reduced complexity in some special cases. I think that at least the construction of the reproducing kernel in the affine coordinates of the big cell of the flag manifold can be implemented relatively efficiently by polynomial multiplication using a multidimensional fast Fourier transform.