In his comment to the question Algebraic numbers and the complex projective line minus three points JSE says that for algebro-geometric objects defined over the complex numbers "in practice, 'the moduli space of Xes is zero-dimensional' is the same as 'all Xes can be defined over number fields'," so that for example "rigid things like finite etale covers of a fixed curve or elliptic curves with complex multiplication are defined over number fields."
Is it reasonable to fill that in by saying a zero dimensional moduli space sharply constrains the action on Xes of the absolute Galois group of the rationals, so that many field automorphisms of $\mathbb{C}$ are likely to leave a given X unchanged (up to isomorphism), and that large group of field automorphisms is likely to have a finite extension of $\mathbb{Q}$ as fixed field (the moduli field of that given X)? Does it even imply the orbit of each given X is finite?