When is a homogeneous space a variety? Let $G$ be a Lie group and let $H$ be a closed subgroup of $G$.  Then $G/H$ may not be a group, but it will be a homogeneous space for $G$ with stabilizers conjugate to $H$.  Sometimes, this is a variety, for instance, when $G$ is a complex reductive group and $H$ is a complex subgroup, and it will even be projective when $H$ is parabolic (by definition).
However, when we take real Lie groups, the situation seems more subtle.  For instance, if $G=Sp(g,\mathbb{R})$ and $H=U(g)$, then $G/H$ is the Siegel upper half space, which is an (analytic) open set in a variety, namely, the variety is the space of symmetric $g\times g$ matrices and the open set is given by the ones with positive imaginary part.  Similarly, many constructions in Hodge theory, particularly that of period domains, end up coming from real Lie groups, and so may be varieties, open subsets of varieties, or not varieties at all a priori.  Clearly, the quotient being even dimensional is a necessary condition, but I'd be surprised if it were sufficient.
So the first part of the question is

When is a homogeneous space (an open subset of) a variety?

Now, additionally, these period domains often can be quotiented by a discrete (I believe Griffiths says arithmetic) subgroup of the original group to actually get a variety.  For instance (if I'm understanding right) if we take $\mathfrak{h}_g=Sp(g)/U(g)$ above, we can quotient further by $Sp(g,\mathbb{Z})$ and this gives us $\mathcal{A}_g$, the moduli space of abelian varieties.

When is there a discrete subgroup that we can take a further quotient by to get a variety?

 A: I'll try to answer both questions, though I will change the first question somewhat.  Let's work in the setting of a real reductive algebraic group $G$ and a closed subgroup $H \subset G$.  
Your first question asks when $G/H$ is an open subset of some (presumably complex) variety.  I think that this question should be modified in a few ways.
You can't really say that $G/H$ "is a subset" of a variety, since $G/H$ is not a priori endowed with a complex structure.  So you need a bit more data to go with the question -- a complex structure on the homogeneous space $G/H$.  Such a complex structure can be given by an embedding of the circle group $U(1)$ as a subgroup of the center of $H$.  Let $\phi: U(1) \rightarrow G$ be such an embedding, and let $\iota = \phi(i)$ be the image of $e^{pi i} \in U(1)$ under this map.  Such an embedding yields an integrable complex structure on the real manifold $G/H$, I believe (though I haven't seen this stated in this degree of generality).
So now one can ask if $G/H$, endowed with such a complex structure, is an open subset of a complex algebraic variety.  But again, I have some objection to this question -- it's not really the right one to ask.  Indeed, it's very interesting when one finds that some quotients $\Gamma \backslash G /H$ are (quasiprojective) varieties -- but such quotients are not obtained as quotients in a category of varieties, from $G/H$ to $\Gamma \backslash G / H$.  They are complex analytic quotients, but not quotient varieties in any sense that I know.  
So what's the point of knowing whether $G/H$ is an open subset of a variety?  Really, one needs to know properties of $G/H$ as a Riemannian manifold and complex analytic space (e.g. curvature, whether it's a Stein space).  That's the most important thing!
As Kevin Buzzard and his commentators note, under the assumption that $G$ comes from a reductive group over $Q$, and under the assumption that $H$ is a maximal compact subgroup of $G$, and under the assumption that there is a "Shimura datum" giving the quotient $G/H$ a complex structure, the quotient $G/H$ is a period domain for Hodge structures, and the quotients $\Gamma \backslash G / H$ are quasiprojective varieties when $\Gamma$ is an arithmetic subgroup of $G$.  
But these are quite strong conditions, on $G$ and on $H$!  I have also wondered about other situations when $X = \Gamma \backslash G / H$ might have a natural structure of a quasiprojective variety.  A general technique to prove such a thing is to use a differential-geometric argument.  A great theorem along this line is due to Mok-Zhong (Compactifying complete Kähler-Einstein manifolds of finite topological type and bounded curvature, Ann. of Math 1989).  The theorem, as quoted from MathSciNet, reads:
"Let $X$ be a complex manifold of finite topological type. Let $g$ be a complete Kähler metric on $X$ of finite volume and negative Ricci curvature. Suppose furthermore that the sectional curvatures are bounded. Then $X$ is biholomorphic to a Zariski-open subset $X'$ of a projective algebraic variety $M$."
Such results can be applied to prove quasiprojectivity of Shimura varieties of Hodge type.  I believe I first learned this by reading J. Milne's notes on Shimura varieties.
I tried once to apply this to an arithmetic quotient of $G/H$, where $H$ was a bit smaller than a maximal compact (when $G/H$ was the twistor covering of a quaternionic symmetric space) -- I couldn't prove Mok-Zhong's conditions for quasiprojectivity, and I still don't know whether such quotients are quasiprojective.  
A: Part 2: Baily-Borel! At least if $G$ is the real points of, say, a reductive algebraic group, and $H$ is a maximal compact subgroup, and EDIT furthermore if $G/H$ admits a complex structure (I think this is equivalent to saying that $G/H$ is a bounded symmetric domain; note that this rules out e.g. $G=SL_n(\mathbf{R})$ for $n>2$). I guess admitting a complex structure is a necessary condition for being a variety though!
In their seminal Annals paper, Baily and Borel construct sufficiently many theta functions on the quotient of a bounded symmetric domain by an arithmetic subgroup (i.e. "$G(\mathbf{Z})$" (this makes sense up to some finite error if $G$ is a reductive algebraic group over the rationals)) that they can embed the quotient into a big projective space, giving the quotient the structure of a quasi-projective variety over the complexes. This is general enough to explain the symplectic group example you give in the question, for example.
Deligne then went on, axiomatising work of Shimura, to show that if furthermore $G$ satisfied certain axioms, then all of this would go through through over a number field. See Deligne's "Travaux de Shimura" and his article on Shimura varieties in the Corvallis proceedings. This explains why the moduli space of princ polarized ab vars is a variety over the rationals, for example.
