See Geometric Invariant Theory by Mumford, Fogarty, and Kirwan.
If $G$ is reductive acting on a variety $X$, there will be Zariski open subsets $X^{ss}$ and $X^s$ consisting of semistable and stable points for the action, with $X^s \subseteq X^{ss} \subseteq X$. The quotients $X^{ss}/G$ and $X^s/G$ will exist as varieties. The quotient $X^{ss}/G$ is a categorical quotient, and the quotient $X^s/G$ is a geometric quotient (it satisfies many additional nice properties, like giving $X^s \to X^s/G$ the structure of a principal bundle).
If the action is scheme-theoretically proper and free, this may be enough to guarantee that $X = X^s$ (when the action is free, $X^s = X^{ss}$ is automatic). There is a lot of material in the above reference that is devoted to analysis of stability, so I am almost certain that this situation is treated in it.
When $G$ is not reductive, then extremely bad things can happen, as in Francesco's answer above.