Rigidity, moduli space, and moduli field

In his comment to the question Algebraic numbers and the complex projective line minus three points JSE says that for algebro-geometric objects defined over the complex numbers "in practice, 'the moduli space of Xes is zero-dimensional' is the same as 'all Xes can be defined over number fields'," so that for example "rigid things like finite etale covers of a fixed curve or elliptic curves with complex multiplication are defined over number fields."

Is it reasonable to fill that in by saying a zero dimensional moduli space sharply constrains the action on Xes of the absolute Galois group of the rationals, so that many field automorphisms of $\mathbb{C}$ are likely to leave a given X unchanged (up to isomorphism), and that large group of field automorphisms is likely to have a finite extension of $\mathbb{Q}$ as fixed field (the moduli field of that given X)? Does it even imply the orbit of each given X is finite?

-
Perhaps include a link alongside the question number? In the new software you can even get the link for particular comments if you'd like. – Asaf Karagila Jul 21 '13 at 23:09
Also, following your question number, it shows that the number is actually the answer's id. The question's number is 137108. The comment can be found in this direct link. (Generally, the timestamp of a comment is a direct link to the comment.) – Asaf Karagila Jul 21 '13 at 23:12
JSE's statement means just that for a locally finite type moduli scheme over an algebraically closed field $F$, if there are no nontrivial dual-number deformations at $F$-points then for any $F'/F$ all $F'$-structures arise over $F$. But in the absence of precise hypotheses one can be led to nonsense. Consider an elliptic curve $E$ over $\mathbf{C}$ with $j(E)$ not algebraic. Pointed connected finite etale covers of $E$ are "rigid" but not defined over a number field as abstract curves. This is unsurprising because one hasn't even defined a moduli scheme over $\overline{\mathbf{Q}}$! – user36938 Jul 22 '13 at 4:04
@user36938 Of course what I wrote (like what JSE wrote) is not precise. But does your first observation capture all the scope of his remark? And is it somehow trivial from the definition of dual number deformation? I had guessed it was rather a Galois theoretic point. Roughly how is it proved? – Colin McLarty Jul 22 '13 at 10:05
Another principle you (& JSE?) may prefer: if $F'/F$ is an extension of algebraically closed fields, $X$ is a "finite type algebro-geometric structure" over $F'$ with no nontrivial automorphisms, and $g^*(X)\simeq X$ for all $g\in G={\rm{Aut}}(F'/F)$ then $X$ descends to $F$. This is true if $X$ arises from an $F'$-point of a separated Artin stack $M$ of finite type over $F$. Indeed, it is a theorem that the locus of "rigid" geometric points of such an $M$ is Zariski-open and an algebraic space, and if $U$ is an algebraic space of finite type over $F$ then $U(F')^G=U(F)$ (via work of Knutson). – user36938 Jul 22 '13 at 11:15