# Branching laws for smooth representations

Let $$E / F$$ be a quadratic extension of nonarchimedean local fields (characteristic 0 if it matters), and $$\pi$$ an irreducible infinite-dimensional smooth representation of $$GL_2(E)$$. Let $$B$$ be the upper-triangular Borel of $$GL_2$$. I'd like to know: do we always have $$\operatorname{dim} Hom_{B(F)}(\pi, \mathbf{C}) = 1 ?$$ I know how to construct a non-zero element in this space, so my concern is to prove its dimension is $$\le 1$$, i.e. to show that $$(GL_2(E), B(F))$$ is a Gelfand pair.

Here's why I think it should be true. Firstly, the analogous result is true if we replace the field extension $$E$$ with $$F \oplus F$$; this is the main result of the Harris--Scholl paper cited below. Secondly, it's true if $$\pi$$ is discrete-series: by Frobenius reciprocity we have $$Hom_{B(F)}(\pi, \mathbf{C}) = Hom_{GL_2(F)}(\pi, Ind_{B(F)}^{GL_2(F)}(\mathbf{C}))$$. The representation $$Ind_{B(F)}^{GL_2(F)}(\mathbf{C})$$ has a trivial subrepresentation and the quotient is the Steinberg representation $$\mathrm{St}$$, so we have a left-exact seq $$0 \to Hom_{GL_2(F)}(\pi, \mathbf{C}) \to Hom_{B(F)}(\pi, \mathbf{C}) \to Hom_{GL_2(F)}(\pi, \mathrm{St}).$$ From the results in Prasad's 1992 paper cited below, exactly one of $$Hom_{GL_2(F)}(\pi, \mathbf{C})$$ and $$Hom_{GL_2(F)}(\pi, \mathrm{St})$$ is 1-dimensional and the other is zero, so we are done.

However, if $$\pi$$ is a principal series, we have the same exact sequence but it can happen that $$Hom_{GL_2(F)}(\pi, \mathrm{St})$$ and $$Hom_{GL_2(F)}(\pi, \mathbf{C})$$ are both non-zero (this occurs iff $$\pi$$ is the normalised induction of a pair of characters of $$E^\times$$ which are distinct, and both trivial on $$F^\times$$). I tried to bash out this case via Mackey theory, but I couldn't get it to work.

Harris, M.; Scholl, A. J., A note on trilinear forms for reducible representations and Beilinson’s conjectures., J. Eur. Math. Soc. (JEMS) 3, No. 1, 93-104 (2001). Preprint version Published version

Prasad, Dipendra, Invariant forms for representations of $$\text{GL}_2$$ over a local field, Am. J. Math. 114, No. 6, 1317-1363 (1992).

• If $E/F$ is a finite field extension (not characteristic 2) and $\pi$ is a generic, principal, non-Steinberg representation, that is not the normalised induction of a pair of characters of $E^\times$ which are distinct, and both trivial on $F^\times$, then I think that $(\pi, 1)_{B(F)}=0$. Are you sure that over local fields the hom-space is at least one dimensional? And if $\pi$ is the normalised induction ... then $(\pi, 1)_{B(F)}=2$, so I would guess that over local fields too $dim\ Hom=2$. Dec 18, 2020 at 11:07
• Yes, I'm sure it's non-zero, because one can construct a canonical element of this space using the Asai zeta integral. Over a finite field the reps are all semisimple, so the exact seq is obviously exact on the right; however smooth reps over local fields are not semisimple in general, so the sequence continues into a long exact sequence of Ext groups, and the content of my question is whether or not the map $Hom_{GL_2(F)}(\pi, \mathrm{St}) \to Ext^1_{GL_2(F)}(\pi, \mathbb{C})$ is non-zero. Dec 18, 2020 at 12:52
You can approach the problem via the mirabolic subgroup $$P_2(F)\subset B_2(F)$$. First we can restrict to $$\pi$$ with central character trivial on $$F^\times$$. Then you want to know if in this situation $$Hom_{P_2(F)}(\pi,1)$$ is of dimension at most $$1$$. This is indeed the case for unitary representations for example by [Matringe, Pacific Journal 2014, Proposition 2.5]. In particular this seems to take care of the remaining case you had of representations of the form $$\chi_1\times \chi_2$$ with $$(\chi_i)_{|F^\times}=1$$, which are clearly unitary as both $$\chi_i$$'s have to be unitary in this case.