Let $B$ be an indefinite quaternion algebra over the rationals, let $G$ be the reductive algebraic group defined by $G(A) = (B\otimes A)^*$ for ${\bf Q}$-algebras $A$; hence $G({\bf R}) = GL_2({\bf R})$ in particular. Then $(G, {\bf H}^{\pm})$ is a Shimura datum (${\bf H}^{\pm}$ is a the union of lower and upper half planes) and it embeds into $(GSp_4, {\bf H_2}^{\pm})$ (Shimura datum defining $A_2$, the moduli space of principally polarized abelian varieties of dimension 2). This induces an embedding of the Shimura curve defined by $(G, {\bf H}^{\pm})$ into $A_2$. On the modular level, this map is just forgetting quaternioninc multiplication. I am aware that there is a slight issue of connected components but this seems irrelevant for what I am going to ask...

My question is what the centralizer of $G$ in $GSp_{2g}$ would be. This is what I am confused about... Logically, this should be $B^*$ because this centralizer represents endomorphisms of the corresponding abelian surface, which in this case case is $B$. However, this is not possible since this centralizer must be compact over ${\rm R}$ (modulo centre) which of course $B^*$ is not...

This is really confsing, I am clearly missing something.... can anyone help? Many thanks in advance!