It is true for example if $X$ is the Poincare half-plane with the Poincare metric.

Hi Chris, so I just had a look at Rubin's argument and he assumes from the outset the $E$ can be defined over K, so this means that $K$ has class number one. His proof does not seem to generalize to the more general setup of de Shalit. In fact, I suspect de Shalit statement to be false.

Hi Chris thanks for the reference, I'll look at it tomorrow. Have a look also at the proof of (iv) of Lemma 21 of his paper "congruences for special values of L-functions...", where he takes into account the roots of unity in $K$ (which is assumed to have class number one, so included in the set up of my question).