Let $Y$ be a smooth projective curve defined over a number field $K$, and let $B$ be a subset of $Y(K)$. It is known that the isomorphism class of a branched cover $f:X(\Bbb{C})\rightarrow Y(\Bbb{C})$ which is unramified outside $B$ is determined by its monodromy representation $\rho:\pi_1(Y(\Bbb{C})-B)\rightarrow S_d$ where $d=\deg f$. My question is what can be said about the field of definition or the field of moduli of the smooth projective curve $X$ based on the monodromy of $f:X(\Bbb{C})\rightarrow Y(\Bbb{C})$? Are there bounds on its degree as an extension of $K$? I know that the monodromy group is invariant under the action of ${\rm{Gal}}(\bar{K}/K)$; so denoting the image of $\rho:\pi_1(Y(\Bbb{C})-B)\rightarrow S_d$ by $H$ (which is a transitive subgroup of $S_d$), the number of group isomorphisms $\pi_1(Y(\Bbb{C})-B)\stackrel{\cong}{\rightarrow}H$ should be a bound for the number of non-isomorphic morphisms in the Galois orbit $\{f^\sigma:X^\sigma\rightarrow Y\}_{\sigma\in{\rm{Gal}}(\bar{K}/K)}$. I wonder if anything more specific/stronger is known in the literature.
Update: Following the answer by @WillChen, I realized that the previous upper bound on the size of Galois orbit in my answer can be changed from the number of isomorphisms $\pi_1(Y(\Bbb{C})-B)\stackrel{\cong}{\rightarrow}H$ to that number divided by $N_{S_d}(H)$; because the monodromy, as a subgroup of $S_d$, is determined up to conjugacy -- there is no canonical way of numbering the elements of a regular fiber of $f:X(\Bbb{C})\rightarrow Y(\Bbb{C})$.
I also came across this article Fields of Definition of Some Three Point Ramified Field Extensions from 1994 which computes the field of definition in some examples with $|B|=3$.