Behaviour of Hilbert functions Let $G$ be a complex simple reductive group. Then the set of isomorphy classes $Irr G$ is isomorphic to the set of dominant weights $\Lambda_+$ in the weight lattice of the maximal torus of a Borel subgroup of $G$. Further, $\Lambda_+$ is isomorphic to $\mathbb{N}_0^\ell$ for some integer $\ell$. Thus we can write $\rho = n_1\lambda_1 + ... + n_{\ell}\lambda_{\ell} = \underline{n}$ for any irreducible representation $\rho\colon G \to End(V_{\rho})$.
Now let $R$ be a ring with $G$-action, $M$ an $R$-$G$-module with isotypic decomposition $M \cong \bigoplus_{\underline{n} \in \mathbb{N}_0^{\ell}} M_{\underline{n}}\otimes_{\mathbb{C}}  V_{\underline{n}}$. Then one defines the Hilbert function of $M$ as
$h(\underline{n}) := rk (M_{\underline{n}})$. Further, consider the function $P(M,\underline{z}) := \sum_{\underline{n} \in D} h(\underline{n})z_1^{n_1}...z_{\ell}^{n_\ell}$ for some finite subset $D \subset \mathbb{N}_0^{\ell}$.
What is the behaviour of $P(M,\underline{z})$ if $D$ is sufficiently large? Is it a rational function?
If $M'$ is a submodule of $M$, how are $P(M',\underline{z})$ and $P(M,\underline{z})$ or $h'$ and $h$ related? Does $\frac{h'(\underline{n})}{h(\underline{n})}$ converge for $\underline{n} \not\in D$ as $D$ becomes large? Which assumptions on $R$ and $M$ are necessary? What is the limit?
I am especially interested in the case $G = Sl_2$ (i.e. $\ell = 1$) but I would also like to know the answer in the general case.
 A: I am not sure if I completely understand all of the assumptions (especially the role of $R$), but let me give it a try.
Let me assume that $R$ is finitely generated over the complex numbers. Since you want to assume that $M$ is also a finitely generated $R$-algebra, let's just throw out $R$ for now and consider $M$, which is a finitely generated C-algebra. I will also assume that the multiplication in $M$ is compatible with the $G$-action.
It seems you've already chosen a set of positive roots, so let $B$ be a Borel subgroup corresponding to these positive roots, and let $U$ be its unipotent radical. The $U$-invariants of $M$ are just the highest weight vectors and we get a decomposition
$\displaystyle M^U = \bigoplus_{\underline{n} \in \Lambda_+} M_{\underline{n}}$
where the dimension of $M_{\underline{n}}$ is the same as the one you are considering. Since the multiplication in $M$ is compatible with $G$, $M^U$ is actually multigraded by $\Lambda_+$. So the function $P(M,\underline{z})$ is a rational function in $\underline{z}$ if we know that $M^U$ is finitely generated (as an algebra).
A general fact: if reductive $G$ acts on a finitely generated algebra $A$, then the $G$-invariants $A^G$ is also finitely generated. Let ${\bf C}[G/U]$ be the coordinate ring of the coset space $G/U$ (which is a quasi-projective variety). Now take $A = M \otimes {\bf C}[G/U]$ where $G$ acts diagonally. As a $G$-module, ${\bf C}[G/U]$ is a direct sum of all irreducible representations of $G$, each appearing with multiplicity 1. Hence the $G$-invariants of $A$ can be identified with the $U$-invariants of $M$, so $M^U$ is finitely generated.
(Warning: in general, if the action of $U$ did not come from an action of $G$, the invariant subalgebra need not be finitely generated)
I can't comment on the second part about submodules. I've never seen a situation where one takes a quotient of the Hilbert series.
