Centralizer of Shimura datum defining a Shimura curve in $A_2$

Let $B$ be an indefinite quaternion algebra over the rationals, let $G$ be the reductive algebraic group defined by $G(A) = (B\otimes A)^*$ for ${\bf Q}$-algebras $A$; hence $G({\bf R}) = GL_2({\bf R})$ in particular. Then $(G, {\bf H}^{\pm})$ is a Shimura datum (${\bf H}^{\pm}$ is a the union of lower and upper half planes) and it embeds into $(GSp_4, {\bf H_2}^{\pm})$ (Shimura datum defining $A_2$, the moduli space of principally polarized abelian varieties of dimension 2). This induces an embedding of the Shimura curve defined by $(G, {\bf H}^{\pm})$ into $A_2$. On the modular level, this map is just forgetting quaternioninc multiplication. I am aware that there is a slight issue of connected components but this seems irrelevant for what I am going to ask...

My question is what the centralizer of $G$ in $GSp_{2g}$ would be. This is what I am confused about... Logically, this should be $B^*$ because this centralizer represents endomorphisms of the corresponding abelian surface, which in this case case is $B$. However, this is not possible since this centralizer must be compact over ${\rm R}$ (modulo centre) which of course $B^*$ is not...

This is really confsing, I am clearly missing something.... can anyone help? Many thanks in advance!

• Couldn't it be the endomorphisms that commute with the Rosatti involution or something like that? – Aurel Sep 29 '16 at 12:23
• yes, I thought about this - it has to have something to do with the polarization... I think that the centralizer ought to be some torus, compact over ${\bf R}$ (like some imaginary quadratic field modulo ${\bf G}_m$...), but which one? – user42721 Sep 29 '16 at 13:11
• I don't see why you'd expect the centralizer to be the endomorphisms of the abelian surface, because why would an endomorphism commute with the action of B? – znt Sep 29 '16 at 19:18
• I may be wrong, but I thought that the centralizer of $G$ in $GSp_{4}$ represented endomorphisms of the polarised Hodge structure whose Mumford-Tate group is $G$ and I thought that it should be $B^*$, but maybe the fact that the endomorphisms are required to be polarized makes it smaller.... – user42721 Sep 30 '16 at 6:01
• This is a purely algebro-geometric question, right? One can write down explicitly the mapping $B^\times \to GSp_4$ and then it's just a computation of a centraliser in an algebraic group -- the context of Shimura varieties, endomorphisms of ab vars, etc is not necessary for the question. So might clear things up a bit if you describe explicitly what the map $B^\times \to GSp_4$ in this problem actually is. – David Loeffler Sep 30 '16 at 8:11

There is a piece of data missing from the question. As explained in this question, in order to get the embedding of Shimura data, you have to choose an element $\gamma \in B^\times$ such that $\gamma^* = -\gamma$ and $\gamma^2$ is a negative rational number. Then there is a $\mathbf{Q}$-bilinear form on $B$ preserved by left multiplication up to scaling, which gives an embedding $B^\times \hookrightarrow GSp_4$.
It's now an easy exercise to work out the endomorphisms. The endomorphisms of $B$ as a $Q$-vector space commuting with the left $B$-action are just $B$, acting on the right via the involution; but the requirement to preserve the symplectic form restricts you to things which commute with $\gamma$, so you just get the imaginary quadratic field $K$ inside $B$ generated by $\gamma$. Hence the centralizer of $B^\times$ in $GSp_4$ is $K^\times$.