I would say there are two issues: first of all, the good thing making the above machinery work is that the moduli problem you look at, represented by $Y_\text{nonsplit}(n)$, works for all quadratic fields at once. In other words, you are interested in looking at all possible curves with CM by some imaginary quadratic field and you end up looking at points always in the same moduli space. This is not the case for Hilbert-Blumenthal surfaces with real multiplication by a real quadratic $F$, who do possess a moduli space (stack, indeed) $\mathfrak{M}_F$ which heavily depends upon $F$.

But suppose this is only a technical issue, either by constructing some monstrous $\mathfrak{M}$ parametrizing abelian surfaces with al multiplication by some quadratic field not specified in the moduli problem; or by hoping, in your "pipe dream" that one proves that there are infinitely many $F$ with at least one element in $\mathfrak{M}_F(\mathbb{Q})$. Then a major (and, to my knowledge and understanding, not a merely technical one) problem is the following: the crucial step in the CM procedure, as you said, is the implication (now $K$ is imaginary quadratic) $$ h(K)=1\Leftrightarrow j(E_K)\in\mathbb{Q} $$ (it is indeed integral, but that won't matter, here) where $E_K$ is some/any elliptic curve with CM by $K$. This is false for real multiplication: to convince yourself of the failure of one implication, observe that there are elliptic curves with real multiplication by $\mathbb{Q}$ (what else?) which are not defined over the rationals although $h(\mathbb{Q})=1$. Likewise, the field of definition of the abelian surface tells you almost nohing on the arithmetic of the field of real multiplication, unlike the CM case, preventing you from reducing the class number problem to counting varieties (with bonus structure) defined over $\mathbb{Q}$ - i.e. rational points on some moduli space.