Suppose G is an affine algebraic group acting linearly on a vector space V. A point v∈V is stable if the orbit Gv is closed and v is regular (the dimension of the stabilizer of v is locally constant, or equivalently, locally minimum). I would really like to say this is equivalent to the orbit Gv being closed and not being in the closure of another orbit.
Since the orbit of a regular point has locally maximum dimension, it can't be in the closure of another orbit. But is the converse true? If a point is not in the closure of an orbit larger than its own, is it regular?
The answer is no ... we have to throw in some hypotheses. If you consider the action of Ga on A2 given by t(x,y)=(x,tx+y), then all the orbits are closed, but points of the form (0,y) are irregular. So let's throw in the hypothesis that G is linearly reductive. I feel like we might also want to insist that v≠0, but I'm not sure about that.
Linearly reductive seems like a strange hypothesis, so feel free to modify it. I was thinking that you could somehow show that if v is not in the closure of a larger-dimensional orbit, then span(Gv) would be an invariant subspace with no complement, but I haven't been able to get this argument to work.