Let us work over $\mathbb{C}$ for the moment.
Assume we are given a real quadratic field $K$ with ring of integers $\mathcal{O}_K$.
$\mathbf{Question:}$ Is there a smooth projective curve $C$ of genus $g=2$ such that $End(Jac(C))$ is a non-maximal order in $K$, that is $End(Jac(C))=\mathcal{O}_{K,f}=\mathbb{Z}+f\mathcal{O}_K$ for some $f>1$, especially $\neq \mathcal{O}_K$? If yes can we find such a curve for every $K$ and every $f>1$?
Yes, for every quadratic ring of discriminant $f^2 D$ (where $D$ is the discriminant of $K$ in your notation), there's a Humbert surface's worth of such Jacobians. See David Gruenewald's thesis (available at http://echidna.maths.usyd.edu.au/~davidg/thesis.pdf) and the references there for calculations of Humbert surfaces. The double cover, which is the Hilbert modular surface, can also be computed in various ways. See https://arxiv.org/abs/1209.3527 for a method using elliptic K3 surfaces (for fundamental discriminants) and https://arxiv.org/abs/1412.2849 for non-fundamental discriminants if K is replaced by the disc 1 quadratic algebra $\mathbf{Q} \oplus \mathbf{Q}$.
