If $X$ is projective, the Bialynicki-Birula decomposition is filterable; this means that there is a filtration $Z_1 \subseteq Z_2 \subseteq \cdots \subseteq Z_r = X$ by closed subsets, such each difference $Z_i \smallsetminus Z_{i-1}$ is a piece of the Bialynicki-Birula decomposition (Białynicki-Birula, Some properties of the decompositions of algebraic varieties determined by actions of a torus. Bull. Acad. Polon. Sci. Sér. Sci. Math. Astronom. Phys. 24 (1976), no. 9, 667–674). This implies that your relation is antisymmetric.
[Edit] When $X$ is only quasi-projective, the Białynicki-Birula decomposition does not exist in general, even if $X^{{\mathbb C}^*}$ is proper. It does exists when $\lim_{t \to 0}t \cdot x$ exists for any $x \in X$. I would guess that in this case it is still filterable, and that you can prove this by choosing a smooth equivariant compactification.
