For any $a = [a_0; \dots; a_n]\in \mathbb{P}^n(\mathbb{Q})$, the corresponding Galois group $G_a$ of $f(X) = a_n X^n + \cdots + a_1 X + a_0\in \mathbb{Q}[X]$ is a subgroup of $S_n$. (I'm interested primarily in the case where $f$ is irreducible here, but we can take $G_a = \text{Aut}(k/\mathbb{Q})$ for the splitting field $k$ of $f$ regardless.) For a given group $G \subset S_n$, what exactly can be said about the size of the set of points $a$ with $G_a = G$? The group $G = S_n$, for example, occurs exactly when the resolvent of $f$ is irreducible, and this happens for $a$ in a Zariskidense set by Hilbert's irreducibility theorem. Generalizing from that, is there a specific sense in which $A(X) = \{a\in \mathbb{P}^n(\mathbb{Q}):\, G_a\subset X\}$ has $A(H)\subset A(G)$ "small" in $A(G)$ for $H < G$? That is, is there a result invoking some notion like the dimension of a variety (although we're necessarily working over $\mathbb{Q}$ here, or at least some other ground field that isn't algebraically closed), a thin set in the sense of Serre, etc. that makes this vague idea of "smallness" more precise?

$\begingroup$ Hope this is appropriate for this board; I originally posted it on math.stackexchange a while back but didn't get any response. $\endgroup$– anomalyDec 10, 2021 at 1:23

3$\begingroup$ Seems fine to me! But even if it's been a while, it's still worth linking to the original question (and vice versa). $\endgroup$– Will SawinDec 10, 2021 at 1:30

1$\begingroup$ @WillSawin: Thanks, good idea: math.stackexchange.com/questions/4187103/… $\endgroup$– anomalyDec 10, 2021 at 1:33
2 Answers
Every Galois group is Zariski dense, so that's not a very exciting measurement. In fact, for any degree n field, minimal polynomials of generators of that field are Zariski dense. This follows from the fact that the map taking an element to its minimal polynomial is finitetoone, thus sends Zariski dense sets to Zariski dense sets. (Or, more generally, for any rank n étale algebra, characteristic polynomials of nondegenerate elements are Zariski dense.)
Typically the more precise question one asks is the number of such polynomials of height up to $H$, or really the asymptotics as a function of $H$. Taking Weil height, this is equivalent to asking the number of polynomials $\sum_{i=0}^n a_i X^i \in \mathbb Z[x]$ with a given Galois group, where the $a_i$ are relatively prime and $a_i<H$. (In other words, the Weil height is the maximum coefficient, after clearing denominators and removing common factors.)
This is almost an active research question. The almost is because researchers usually set $a_n=1$. For this problem, the best results known are in the recent work of Bhargava, and according to that paper the same methods work for the varying $a_n$ case, but you will not find exact estimates for that version in the paper.
The basic shape is that, for each Galois group, the number of such polynomials should be proportional to $H$, with the actual power varying greatly with the choice of Galois group, and only upper and lower bounds known for most groups rather than the exact figure.
Just to add one thing to Will Sawin's answer:
The set of points $a \in \mathbb{P}^n(\mathbb{Q})$ with $G_a \ne S_n$ is a thin set in the sense of Serre; proving this is one of the standard modern analytic approaches to Hilbert's Irreducibility theorem, e.g. in Serre's Lectures on the MordellWeil theorem. Your sets $A(H)$ are all subsets of this, and so are also thin (and in fact this is usuall proved first in this approach, e.g. with resolvent polynomials).
(As Will notes, often people fix the leading cofficient to be 1, that is, they work with $\mathbb{A}^n$ rather than $\mathbb{P}^n$, but this doesn't affect thinness.)
The large sieve inequality for thin sets then gives an upper bound on the number of points of $A(H)$ of bounded height, which shows that the proportion of such points goes to 0 as the height goes to infinity. However, these bounds are not sharp, because the sets $A(H)$ are "special" sorts of thin sets. Will's answer discusses the quantitative state of current research on this problem.

$\begingroup$ Great, thanks! At the very least, it sounds like the right formulation of the question is considering asymptomic properties of $f(m) = \#\{a\in \mathbb{P}^n:\, G_a = G, H(a)\leq m\}$ (or $\mathbb{A}^n$, which should only affect the bookeeeping) for fixed subgroups $G\subset S_n$. $\endgroup$– anomalyDec 10, 2021 at 19:13