I haved see Illusie's paper, it is true even in some affine case ($X$ is affine ), but I think it may has obstruction in case of smooth projective curve.

@Mikhail: I mean the maximal commutative subalgebras of endomorpisms of the corresponding abelian varieties of CM-type.

@Mikhail: In my notation E is just your Z, and The PEL type shimura variety I mentioned is exactly the "auxiliary Shimura variety of PEL-type". And My question is about the CM-point of this "auxiliary Shimura variety of PEL-type", In fact I even wonder is there any classification of CM-fields for given PEL-type shimura datum.

So if we require that these two Abelian varieties are ordinary, then the answer will be no ?