Is the tensor product of two infinite dimensional objects in the BGG category $\mathcal{O}$ of a semisimple Lie algebra always not in $\mathcal{O}$?

Question

Let $\mathfrak{g}$ be a finite dimensional complex semisimple Lie algebra (according to a comment of Victor Ostrik, we need to further require that $\mathfrak{g}$ is simple) and we can consider its BGG category $\mathcal{O}$. It is well-known that $\mathcal{O}$ is not closed under tensor product, i.e. take two $\mathfrak{g}$-modules $M$ and $N$, the tensor product $M\otimes N$ is still a $\mathfrak{g}$-module but not necessarily in the category $\mathcal{O}$.

In an answer to this MO question, Jim Humphreys showed that an object $M$ in the category $\mathcal{O}$ is “tensor-closed” (meaning that $M\otimes N$ is in $\mathcal{O}$ whenever $N$ is) if and only if $M$ is finite dimensional. He also wrote a note on it.

My question is: if $M$ and $N$ in $\mathcal{O}$ are both infinite dimensional, then is it always true that $M\otimes N$ is no longer in $\mathcal{O}$? In particular let $L(\lambda)$ and $L(\mu)$ be the unique simple module with highest weights $\lambda$ and $\mu\notin \Lambda^{+}$, is $L(\lambda)\otimes L(\mu)$ in the category $\mathcal{O}$?

Answer 1

The answer to the question here is yes. (More generallly, if $M$ and $N$ lie in $\mathcal(O)$ and are both infinite dimensional, then $M \otimes N \notin \mathcal{O}$.)

The proof is eaiest to write down in case $\mathfrak{g} = \mathfrak{sl}_2$ with $M = L(\lambda)$ and $N=L(\mu)$ with $\lambda, \mu \in \mathbb{Z}$ and $\lambda, \mu \leq -1$; but the idea is similar for other pairs. The main point is that $M \otimes N$ fails to have a formal character (i.e., fails to be of finite Jordan-- Holder length). For example, when $\lambda =\mu = -1$, the weights of $L(\lambda) \otimes L(\mu)$ are $-2, -4, -4, -6,-6,-6, \dots$, so the infinitely many composition factors are those of $M(-2), M(-4), M(-6), \dots$ This is contrary to the finite length of all modules in $\mathcal{O}$.

Note here that $L(\lambda) = M(\lambda)$ is a simple Verma module and similarly $L(\mu)=M(\mu)$. $L(-2) = M(-2)$, etc.

ADDED: The point of the preceding note is to give examples where it's easy to specify the formal character of a simple module without getting into the details of the Kazhdan-Lusztig methods. But I'd also expect the rank 1 case to be sufficient for dealing with the general case: for a suitable simple root, consider how representations of $\mathfrak{g}$ restrict to the rank 1 Lie algebra corresponding to the root, etc. This approach hasn't been thoroughly worked out, however.

Answer 2

Edit: As @ZhaotingWei points out this is wrong. In fact, in my example we seem to have $M\simeq \bigoplus_{n=4}^\infty \Delta(-n\rho)$ which is certainly not in $\mathcal{O}$.

Inspired by @VictorOstrik's answer, I believe (but there may be holes in my logic) that this is a counter example in $\mathfrak{sl}_3$:

Let $\rho$ be the half-sum of positive roots, $s, t$ be the simple reflections in $S_3$, and for $w\in S_3$, $\lambda\in\mathfrak{h}^*$, let $w\cdot \lambda = w(\lambda + \rho) - \rho$ denote the usual 'dot action', then consider $M = L(st\cdot 0) \otimes L(ts\cdot 0)$ (for a visual see page 27 in my paper mentioned above). By weight considerations, it is clear that $M$ will have a highest weight vector $-3\rho$. This is an anti-dominant weight, so the Verma module $\Delta(-3\rho)$ must occur as a subquotient of $M$.

Now, the non-zero weight spaces of $L(st\cdot 0)$ and $L(ts\cdot 0)$ all have dimension 1 (I believe, it's been a while so I may be wrong on this). Working out the formal character of $M$ then shows that $\operatorname{ch} M = \operatorname{ch} \Delta(-3\rho)$, so we must have $M\simeq\Delta(-3\rho)$ which is in $\mathcal{O}$.

Answer 3

I will answer the question in the affirmative for $\mathfrak{sl}_n$, and in general provide a property of the Weyl group that would imply an affirmative answer in general. This property is true in type A, but not for the other rank 2 cases. The other rank 2 Lie algebras are small enough to be dealt with easily.

Let me assume for now that λ is regular and integral. Write $\lambda = x\cdot \lambda'$, where x is in the Weyl group and λ' is dominant. Write ch(L(λ)) as a rational function in reduced form, and let T be the set of positive roots α such that $(1-e^\alpha)$ does not appear in the denominator. Let G be the subgroup generated by all reflections in all elements of T. I claim that if g is in G, then gx≤x in Bruhat order.

Switch to indexing simples and Vermas by the Weyl group. We have $$\operatorname{ch}(L(x))=\sum_y P_{x,y}\operatorname{ch}(M(y)).$$ The claim follows from two simple observations. Firstly, for $(1-e^\alpha)$ to not appear in the denominator, we must have $P_{x,y}+P_{x,sy}=0$, where s is the reflection for the root α. Secondly, that P_x,x=1 and P_x,y=0 unless y≤x.

Now we turn to the question. Consider two simples L(λ) and L(μ). From the above discussion we obtain two Weyl group elements x and y, together with two sets of positve roots T_x and T_y, and groups G_x and G_y generated by the corresponding reflections, so that $G_xx\leq x$ and $G_y y\leq y$. If $L(\lambda)\otimes L(\mu)$ is in category O, then $T_x\cup T_y$ is all positive roots by looking at the character. For L(λ) and L(μ) to be infinite dimensional, we must have that x and y are not the identity.

Now we have a condition on the Weyl group which is necessary for a negative answer. In the symmetric group, subgroups generated by reflections are products of smaller symmetric groups. It is easy to see that for any two proper such subgroups, there is a reflection which is not contained in either. Thus in type A, since every positive root appears in either T_x or T_y, either G_x or G_y is the entire symmetric group, which then contradicts the Bruhat ordering condition as x and y are not the identity.

If we move beyond regular or integral weights, then it becomes even harder to cancel factors in the denominator of the character, so the argument should be easier. Beyond type A, I haven't checked any non-type A Weyl groups of rank at least three to see if my criterion is enough to answer the question.