To elaborate a bit: when we are saying "GW invariants are invariant under small deformations", then we are implicitly already using the Gauss-Manin connection: If $X \to T$ is a family, say over the disc, then it doesn't make sense to identify Gromov-Witten invariants of $X_{t_1}$ and $X_{t_2}$ unless you know which GW-invariants to compare; in other words, given cohomology classes $\gamma_1, \dots, \gamma_n$ and a homology class $\beta$ on $X_{t_1}$, we need to find corresponding classes on $X_{t_2}$. Well, fortunately, this is not a problem, as $X_{t_1} \cong X_{t_2}$ as smooth manifolds, and thus $H^*(X_{t_1}) \cong H^*(X_{t_2})$.This identification is nothing but the Gauss-Manin connection. In particular, when the bases $T$ is more complicated, you need to choose a path from $t_1$ to $t_2$ to obtain the identification; and it will depend on the homotopy class in the path. In particular, when $T$ is not simply-connected, we get a representation of $\pi_1(T)$ on $H^*(X_0)$, i.e. the monodromy action. Since it is pieced together out of identifications $H^*(X_0) \cong H^*(X_{t_1}) \cong H^*(X_{t_2}) \cong \dots$ as above, GW-invariants are invariant under this group action:
\langle \gamma_1, \dots, \gamma_n \rangle_{\beta}^{g,n} =\langle \Phi(\gamma_1), \dots, \Phi(\gamma_n) \rangle_{\Phi(\beta)}^{g,n}for any $\Phi$ in the image of $\pi_1(T) \to \mathrm{Aut} H^*(X, \mathbb{Q})$.(The mapping class group is basically the biggest possible group $\pi_1(T)$ you can obtain this way, i.e. the fundamental group of the moduli space of varieties diffeomorphic to $X$.)
