This question arises from a comment by user nfdc23 on an unrelated recent MO question here. It concerns textbook treatments of what has been called the "Theorem of Kostant-Rosenlicht", stated as Theorem 2 in this 1961 paper by Rosenlicht. (Apparently Kostant had addressed a similar question earlier involving Lie algebras in characteristic 0, though this may be unpublished.) Rosenlicht was mainly concerned with quotient varieties related to the action of a unipotent algebraic group $G$ on a quasi-affine variety $V$ (an open set in an affine variety). By a somewhat complicated induction on the dimension of $G$ (which can just as well be assumed connected), he proved in Theorem 2 that the $G$-orbits in $V$ are all closed. As an immediate corollary, the conjugacy classes in $G$ itself are closed.
Each of the three textbooks called Linear Algebraic Groups includes some version of this theorem: Borel's 1969 Benjamin lecture notes (expanded to a second edition in the Springer GTM series in 1991) stated Rosenlicht's theorem as Proposition 4.10 but with a proof which works only when $V$ is affine. My own GTM book in 1975 included just an exercise (17.8) on the affine case, with an outline of the proof. Then Springer in 1981/1998 treated the affine case briefly in his Proposition 2.4.14.
Is there a natural example of a quasi-affine but not affine variety $V$ to which Rosenlicht's theorem applies in an essential way?
I've found it difficult to place the theorem in the study of algebraic groups and their actions. For example, it doesn't seem to play any role in the basic structure theory of reductive groups.
ADDED: The answers and comments by nsfdc23, YCor, and Friedrich Knop have been very helpful in clarifying the issues here. The implicit answer to my question seems to be that there are no examples of the sort I asked about. Indeed, there seems to be little distance between the affine and quasi-affine formulations. This is less obvious in Rosenlicht's paper, especially because the language he used isn't as flexible as the language of schemes.
Borel apparently saw how to simplify Rosenlicht's approach by invoking the action of $G$ on regular functions, where any finite dimensional $G$-invariant subspace has a fixed point. But he skipped over the shift from an affine to a quasi-affine variety. Perhaps this was due to impatience and also having too many other projects in mind. (I recall the kind of time pressure he faced in 1968 in splicing together two sets of lecture notes on arithmetic groups which differed a lot in notation. I found many minor corrections to be made but didn't share all of them in a timely way; so the published version still has uncorrected pages.)
I don't remember now why I stated my exercise just for affine varieties, since I was usually following Borel's lecture notes. Unfortunately it's too late to ask Borel or Springer how they arrived at their choices. Springer might for example have understood the extra argument needed to deal with the quasi-affine case and decided to avoid it.