So Ulrich and Geordie were right, Tom Braden was the right person to ask and here is what he told me:

The answer is yes, in the case above $X$ is Whitney stratified.

The argument goes roughly as follows. In the paper [K] the following is shown. Let $X$ be an algebraic stratified variety and $ S$  a stratum.   The set $Sing(S )$ of points in $S $ which do not fulfil Whitney's b condition has the structure of a (semi-)variety of dimension strictly lower that $\dim S $. 

Now applied to our situation :
$Sing(S)$ is necessarly $G$-invariant, hence empty. This answers my question.  




[K] ``A Geometric Proof of Existence of Whitney Stratifications’’, Moscow Math. Journ., 5 (2005), no.1, 125—133