Are there some good asymptotic estimations for the number $F(n)$ of nonisomorphic finite groups of size smaller than $n$?
2 Answers
The behaviour of $F(n)$ varies dramatically with the primefactorization of $n$. Typically one gets a large jump in the value of $F(n)$ as $n$ passes the power of a prime, particularly when that prime is equal to $2$.
The first key result $(\dagger)$ in this area (I believe) is due to Higman and Sims:
Theorem Let $p$ be a (fixed) prime number. Define $f(n,p)$ as the number of groups of order $p^n$. Then: $$f(n,p) = p^{(2/27 + o(1))n^3}.$$
(The link above gives a more detailed version of this result.) A result of Laci Pyber can be combined with that of Higman and Sims to give:
Theorem: Let $n=\prod_{i=1}^kp_i^{g_i}$ be a positive integer with the $p_i$ distinct primes. Let $\mu$ be the maximum of the $g_i$. The number of groups of order $n$ is at most $$n^{(2/27+o(1))\mu^2}$$ as $\mu\to\infty$.
The best way in to this area (it seems to me) is to consult Pyber's paper on the subject containing the above result:
Pyber, L. Enumerating finite groups of given order. Ann. of Math. (2) 137 (1993), no. 1, 203–220.
An interesting extra tidbit from that paper is the following:
Conjecture: Almost all finite groups are nilpotent (in the sense that $f^∗_1(n)/f^∗(n)\to 1$ as $n\to\infty$, where $f^∗(n)$ is the number of isomorphism classes of groups of order at most n and $f^∗_1(n)$ is the number of isomorphism classes of nilpotent groups of order at most $n$).
(In other words all counts are dominated by $p$groups.) You should also refer to Derek's answer  apologies to him for not referencing his result!
$(\dagger)$ I said this was a conjecture earlier  my mistake.

$\begingroup$ Minor issue: the OP mentions groups "of size smaller than $n$", but your opening paragraph is allowing also groups of size equal to $n$ in the value of $F(n)$. $\endgroup$ Dec 11, 2013 at 14:11


$\begingroup$ @S.Carnahan  good point, thank you. I have edited accordingly. $\endgroup$ Nov 28, 2014 at 19:27
It is proved in
Holt, D. F., Enumerating perfect groups. J. London Math. Soc. (2) 39 (1989), no. 1, 67–78
that
$n^{2l(n)^2/27−dl(n)} \le F(n) \le n^{l(n)^2/6+l(n)}$
for some constant $d$, where $l(n) = \log_2(n)$. The lower bound is coming from Higman's construction of large numbers of $p$groups of class 2, and the general belief seems to be that the lower bound is close to being the correct number.
I think that there might be better results known now. You could try searching publications of Laszlo Pyber, but I don't have time right now!
Added later: Nick's answer is much more accurate than mine. But, in case it is of any interest, let me add that the main result of the paper I mentioned was an estimate of the number ${\rm perf}(n)$ of finite perfect groups of order at most $n$, which (perhaps surprisingly) is also large, and satisfies:
$n^{l(n)^2/108−cl(n)} \le {\rm perf}(n) \le n^{l(n)^2/48+l(n)}$, where $c=11/36$.