This is a rewrite of a deleted question. I've decided to focus on one particular example mentioned in that question. Below a point means a closed point.
Let $U$ be the set of irreducible non-cuspidal cubic curves in $\mathbb{P}^2(\mathbb{C})$. The group $U=SL_3(\mathbb{C})$ acts on $U$. Does there exist a categorical quotient $U/G$? And if so, is the quotient $U\to U/G$ geometric? Here we use the definition of the categorical and geometric quotient given in \S 0.1 of GIT by Mumford, Fogarty and Kirwan. In particular, we do not require the map $U\to U/G$ to be affine in the definition of the geometric quotient.
Remarks. Observe that $U$ is contained in $X=$ the set of all cubic curves that are not cuspidal and do not have a triple point. (The latter condition is equivalent to the curve not being a union of three lines such that the intersection of all three is nonempty.) $X$ is precisely the semi-stable locus for the action of $G$ on $\mathbb{P}^{10-1}(\mathbb{C})$, the projectivization of the vector space of degree 3 homogeneous polynomials in three variables, linearized via the action on $\mathcal{O}_{\mathbb{P}^9(\mathbb{C})}(1)$. There is a categorical quotient $X\to\mathbb{P}^1(\mathbb{C})$, which is not geometric. Observe that over each point of $\mathbb{P}^1(\mathbb{C})$ there is precisely one orbit inside $U$, in other words, the $G$-action on $U$ (but not on $X$) is closed.
So a more general setting to consider is this: a reductive group $G$ acts on a variety $X$, and the categorical quotient $q:X\to X/G$ exists; $U\subset X$ is an open subvariety that maps surjectively to $X/G$, and the preimage of every point under $(q|U)^{-1}$ is a single orbit. If there are general conditions which ensure the existence of the categorical ro geometric quotient $U/G$ in this situation, I'd be interested to know.
