Say that two structures $\mathfrak{A},\mathfrak{B}$ in finite languages $\Sigma,\Pi$ are parametrically equivalent - and write "$\mathfrak{A}\approx\mathfrak{B}$" - iff they have the same underlying set and the same set of definable-with-parameters relations.

I'm curious about the possible automorphism groups of a parametric equivalence class. Specifically, for a structure $\mathfrak{A}$ let $$\mathsf{AGS}(\mathfrak{A})=\{\operatorname{Aut}(\mathfrak{B}):\mathfrak{B}\approx\mathfrak{A}\}.$$ (It's here that the "same underlying set" condition is crucial: $\mathsf{AGS}(\mathfrak{A})$ is a family of subgroups of the full permutation group of that underlying set, and I am not identifying isomorphic elements of $\mathsf{AGS}(\mathfrak{A})$.)

For example, many rigid structures are parametrically equivalent to structures with $1$-transitively-acting automorphism group. A bit more surprisingly (to me at least), at MSE user Harry West showed that $\mathsf{AGS}(\mathfrak{A})$ need not be upwards-directed; however, the example he constructed was somewhat artificial.

But that's about all I know. In particular, I don't know much at all about computing $\mathsf{AGS}(\mathfrak{A})$ even for relatively simple $\mathfrak{A}$. I'd like to rectify this:

Question: What exactly is $\mathsf{AGS}(\mathbb{Q};+,\times)$? In particular, is it upwards-directed?