So Ulrich and Geordie were right, Tom Braden was the right person to ask and here is what he told me:

The answer is yes, in the case above $X$ is Whitney stratified.

The Argument goes roughly as follows. In the paper [K] the following is shown. Let $X$ be an algebraic stratified variety and $ S$ a stratum. The set $Sing(S )$ of points in $S $ which do not fulfil Whitney's b condition has the structure of a (semi-)variety of dimension strictly lower that $\dim S $.

Now applied to our situation : $Sing(S)$ is necessarly $G$-invariant, hence empty. This answers my question.

[K] ``A Geometric Proof of Existence of Whitney Stratifications’’, Moscow Math. Journ., 5 (2005), no.1, 125—133

So Ulrich and Geordie were right, Tom Braden was the right person to ask and here is what he told me:

The answer is yes, in the case above $X$ is Whitney stratified.

The argument goes roughly as follows. In the paper [K] the following is shown. Let $X$ be an algebraic stratified variety and $ S$ a stratum. The set $Sing(S )$ of points in $S $ which do not fulfil Whitney's b condition has the structure of a (semi-)variety of dimension strictly lower that $\dim S $.

Now applied to our situation : $Sing(S)$ is necessarly $G$-invariant, hence empty. This answers my question.

[K] ``A Geometric Proof of Existence of Whitney Stratifications’’, Moscow Math. Journ., 5 (2005), no.1, 125—133