The following statement seems to be well-known:

Let $X$ be a variety on which an affine algebraic group $H$ acts with finitely many orbits and let $s \in H$ be semisimple. Then $H_s = \{h \in H \mid hsh^{-1}= s \}$ has only finitely many orbits in $X^s = \{ x \in X \mid sx =x \}$.

For example, this is stated without proof in a paper of Spaltenstein (see the last page of [1]). A similar result is mentioned in the Chriss-Ginzburg book [2, Prop. 8.1.17]. However, their proof uses at some point that for $x \in X$ any semisimple element in $H_x = \{h \in H \mid hx = x \}$ lies in a maximal torus of $H_x$. This seems incorrect given that the stabilizer $H_x$ could be disconnected. A similar argument avoiding this issue is given in [3, §5.4] which instead uses the following statement without proof:

A semisimple class of an algebraic group meets a closed subgroup in a union of finitely many semisimple classes of that subgroup.

For certain groups and under some restrictions on the characteristic, this can be deduced from results of Richardson (see for example [4, §3.8]) but this argument doesn't seem to apply in the general setting.

Is there a good reference or a simple proof for either of these two statements that makes no restriction on the algebraic group or the characteristic of the field?

[1] Spaltenstein, N. "On unipotent and nilpotent elements of groups of type E 6." Journal of the London Mathematical Society 2.3 (1983): 413-420.

[2] Chriss, Neil, and Victor Ginzburg. Representation theory and complex geometry. Birkhäuser, 1997.

[3] Kazhdan, David, and George Lusztig. "Proof of the Deligne-Langlands conjecture for Hecke algebras." Inventiones mathematicae 87 (1987): 153-215.

[4] Humphreys, James E. Conjugacy classes in semisimple algebraic groups. American Mathematical Soc., 1995.