The question. Is an asymptotic equivalent known or conjectured for the number $N(d)$ of $\alpha \in \bar{\mathbb{Q}}$ with $h(\alpha) < 1$ and $[\mathbb{Q}(\alpha):\mathbb{Q}] \leq d$?

The rather crude proof of Northcott's theorem bounds $\log{N(d)}$ by $O(d^2)$. On the other hand, upon extracting $\lfloor d/m \rfloor$-th roots from numbers of degree $m$ and logarithmic height $< d/m$, where $m$ is fixed but arbitrarily large, and using a generalization of Schanuel's theorem due to Masser and Vaaler ("Counting algebraic numbers with large height," Trans. Amer. Math. Soc. 2007), it is easily seen that $\log{N(d)} > Ad$ for $d \gg_A 0$ and any given $A < \infty$.

The mentioned paper of Masser and Vaaler determines the dominant term for the opposite count involving points of large height and bounded degree. The problem considered here, involving bounded height and large degree, is likely to be more delicate. The basic lower bound $\log{N(d)} \gg d$ of the previous paragraph is likely to be improved by taking an optimal $m = m(d)$ and examining the error term (rate of convergence) in the Masser-Vaaler estimate. Perhaps such an improvement could point to the right asymptotics for $\log{N(d)}$?

Has this asymptotic been studied?

The answer. Indeed $\log{N(d)} \asymp d^2$, and this is not hard to see modulo a certain irreducibility hypothesis, which at the very least is implied by a conjecture of Poonen (cf. Thm. 1 in Bertini theorems over finite fields). By the Jensen-Mahler formula, the height $h(\alpha)$ of an algebraic number of degree $d$ with minimal polynomial $f \in \mathbb{Z}[x]$ coincides with the normalized logarithmic Mahler measure: $$ h(\alpha) = d^{-1} \log{M(f)} := d^{-1} \int_{S^1} \log{|f(z)|} \, \frac{d\theta}{2\pi}. $$ By the triangle inequality and a term-by-term integration, this is bounded above by $d^{-1} \log{\ell_1(f)}$, where $\ell_1(f)$ is the $L^1$-norm of the vector of coefficients of $f$. Now of course the inequality $\ell_1(f) < e^d$ has $\exp((1-o(1))d^2)$ solutions $f \in \mathbb{Z}[x]$ of $\deg{f} = d$, and it should not be too difficult to show that as many of them are irreducible (perhaps I am underestimating this problem). This supplies the lower bound $\log{N(d)} \geq (1-o(1))d^2$.

Northcott's bound gives $\log{N(d)} < (1+\log{2})d^2 + O(d)$ in the other direction, and we certainly have $\log{N(d)} \asymp d^2$. Apart from verifying the claim on irreducibility, what remains to be seen is whether equality holds in $\log{N(d)} \geq (1-o(1))d^2$. But I suspect this could be a delicate issue.