It seems that when $p>3$ is a prime, then each group of order $p(p^2+1)/2$ is abelian as I checked by Gap for small $p$. Is it true for each $p$? Thanks for your answers

$\begingroup$ When $p=13$, the order is $5\times13\times17$. Now $13\times17\equiv1\pmod5$, which (I think) enables us to conclude that there's a nonabelian group of that order. $\endgroup$ – Gerry Myerson Feb 21 '14 at 21:32

3$\begingroup$ @GerryMyerson: No. The sylow theorems tell us, that there are unique normal subgroups A,B of order 13 and 17. AB is a normal {13,17}Hall subgroup and any 5sylow C is a complement. Now any morphism $C\to Aut(AB) = Aut(A)\times Aut(B)\cong C_{12}\times C_{16}$ must be trivial since 5 divides neither 12 nor 16. Therefore the semidirect product $G=(AB)\rtimes C$ is actually a direct product, i.e. $G=A\times B\times C$ is cyclic. $\endgroup$ – Johannes Hahn Feb 21 '14 at 23:08

5$\begingroup$ @GerryMyerson: Note that $5\times 13\times 17$ is coprime to $\phi(5\times 13\times 17)=4\times 12\times 16$. So by your(!) nice answer to mathoverflow.net/questions/148731/… there is only one group in this situation. One could also use your answer there to check this  find $p$ with $n=p(p^2+1)/2$ squarefree but with $n$ not coprime to $\phi(n)$. The answer with $p=53$ is such an example. $\endgroup$ – Lucia Feb 21 '14 at 23:57

6$\begingroup$ mathoverflow.net/questions/31553 (answer by R. Chapman) gives a characterization of those integers $n$ for which every finite group of order $n$ is abelian. $\endgroup$ – Richard Stanley Feb 22 '14 at 0:20

1$\begingroup$ For convenience, let me add a link to the previous comment, it's here $\endgroup$ – მამუკა ჯიბლაძე Feb 22 '14 at 9:53
The smallest counterexample I could find is for $p=53$; then $(p^2+1)/2=5\times281$ and the cyclic group of order 281 has an automorphism of order 5, so the corresponding semidirect product will not be abelian.
To make up for my faulty comment, here's a counterexample of a somewhat different type: let $p=193$; then $(p^2+1)/2=5^3\times149$, and there is a nonabelian group of order $5^3$ (as there is of order $q^3$ for every prime $q$), so there's a nonabelian group of order $p(p^2+1)/2$.
In fact, the number of primes $p$ below $x$ for which there are only abelian groups of order $p(p^2+1)/2$ is $o(\pi(x))$. Thus for almost all primes there will be nonabelian groups of order $p(p^2+1)/2$, despite the initial numerical evidence.
To see this, we will use the criterion provided in the answer of Robin Chapman to Finite nonabelian groups of odd order (which was pointed out by Richard Stanley in the comments above). One needs $(n,\Phi(n))=1$ if there are only abelian groups of order $n$; here $\Phi(n)$ a variant of the Euler $\phi$function is multiplicative with $\Phi(p^k) =p^k1$. Also, if $n$ is cubefree (not divisible by the cube of any prime), then the condition $(n,\Phi(n))=1$ guarantees that the only groups of order $n$ are abelian.
Suppose first that $p \equiv \pm 2 \pmod 5$. Then $p^2+1$ is a multiple of $5$, and so if $p^2+1$ is divisible by some prime $\ell \equiv 1\pmod{20}$ then we would certainly have that $5$ divides $(n,\Phi(n))$ (writing $n=p(p^2+1)/2$).
Now the density of primes $\ell \equiv 1\pmod{20}$ is $1/8$, and a sieve argument shows that the set of primes $p\le x$ with $p\equiv \pm 2\pmod{5}$ and $p^2+1$ divisible by no such $\ell$ is $O(\pi(x) (\log x)^{1/4})$; (two residue classes $\pmod \ell$ are forbidden for the prime $p$). Thus almost all primes $p\equiv \pm 2 \pmod{5}$ have nonabelian groups of order $p(p^2+1)/2$.
Next consider $p\equiv \pm 5 \pmod{13}$ so that $p^2+1 \equiv 0 \pmod{13}$, and repeat the same argument with primes $\ell \equiv 1\pmod{52}$. And so on. If there is a small prime $r$ with $r(p^2+1)$ then $p^2+1$ must avoid primes $\ell \equiv 1\pmod{4r}$ and there are few such primes $p$. Lastly it could be that $p$ is such that $p^2+1$ does not have any small prime factor $r$; but then the set of such primes $p$ also has small density.
Optimizing this argument one can show that there are at most $O(\pi(x)/\log \log \log x)$ primes for which there are only abelian groups of order $p(p^2+1)/2$. One can also provide a lower bound of this order (but I didn't check all the details here).

$\begingroup$ Concerning the second paragraph: we have $(p^k,p^k1) = 1$, but for $k \geq 3$ there are nonabelian groups of order $p^k$  how does this fit together with what you write? $\endgroup$ – Stefan Kohl Feb 22 '14 at 21:02

$\begingroup$ @StefanKohl: Oh right; also $n$ is divisible at most by squares of primes (as in Chapman's answer). Note that what I wrote was a necessary condition; but I also edited to include the sufficient condition. $\endgroup$ – Lucia Feb 22 '14 at 21:45

1$\begingroup$ Ah, o.k..  I think your bound $O(\pi(x)/\log \log \log x)$ is really nice. $\endgroup$ – Stefan Kohl Feb 22 '14 at 21:56

$\begingroup$ @StefanKohl: Thanks! This is essentially the argument of Erdos, Murty & Murty. $\endgroup$ – Lucia Feb 22 '14 at 22:00
For any constant $C \in \mathbb{N}$, there are infinitely many primes $p$ such that there are at least $C$ nonabelian groups of order $p(p^2+1)/2$:
Let $q \equiv 1$ mod $4$ be a prime, and $k \in \mathbb{N}$. Then the congruence $r^2 \equiv 1$ mod $q^k$ is solvable. For any solution $r$ and any prime $p \equiv r$ mod $q^k$, the number $n := p(p^2+1)/2$ is divisible by $q^k$. Now for large enough $k$ there are more than $C$ nonabelian groups of order $q^k$, and hence also of order $n$. Since by Dirichlet's Prime Number Theorem there are infinitely many primes $p \equiv r$ mod $q^k$, the assertion follows.
Some examples: for $k = 3$ and $q = 5$ we obtain the following primes $p < 10000$:
193, 307, 443, 557, 1193, 1307, 1693, 2557, 2693, 3307, 3557, 3943,
4057, 4943, 5443, 5557, 5693, 5807, 7057, 7193, 7307, 8443, 8693, 8807
(The first is Gerry Myerson's example.)
For $k = 3$ and $q = 13$, we obtain the following examples $p < 1000000$:
239, 13421, 34913, 44179, 52489, 52967, 78853, 87641, 96907, 100823, 118399, 131581,
132059, 136453, 144763, 184309, 197969, 206279, 211151, 232643, 263401, 272189, 290243,
329789, 334183, 342971, 368857, 400093, 404009, 408403, 417191, 422063, 439639, 452821,
456737, 461609, 469919, 487973, 496283, 500677, 549011, 562193, 593429, 597823, 619793,
632497, 637369, 646157, 667649, 672043, 672521, 680831, 711589, 712067, 729643, 733559,
751613, 759923, 764317, 777977, 782371, 791159, 804341, 835099, 852197, 857069, 865379,
869773, 883433, 887827, 891743, 901009, 909319, 914191, 927373, 931289, 936161, 966919,
984017, 993283.
For $k = 6$ and $q = 5$ (and thus $C = 673$), we obtain the following examples $p < 1000000$:
14557, 16693, 79193, 391693, 672943, 733307, 735443, 829193, 858307, 920807, 952057

$\begingroup$ That's very nice. In fact much more is true, and one can show that for almost all primes up to $x$ there will be a nonabelian group of order $p(p^2+1)/2$. So this may be a candidate for the ``phenomenon of eventual counterexamples." $\endgroup$ – Lucia Feb 22 '14 at 18:29

$\begingroup$ @Lucia: How do you conclude that for almost all primes there is such nonabelian group?  The 2 residue classes (mod $q^3$) for primes $q \equiv 1$ mod 4 don't add up to density 1  so probably your argumentation uses cubefree orders $p(p^2+1)/2$? $\endgroup$ – Stefan Kohl Feb 22 '14 at 18:50

$\begingroup$ Right; one should use instead the criterion provided in the answer linked by Richard Stanley's comment. The decrease is very slow, as in the case of the Erdos, Murty & Murty result on the orders for which there is exactly one group of order $n$. I think there'll probably be about $\pi(x)/\log \log \log x$ primes for which there will be only abelian (or even just one cyclic) groups of order $p(p^2+1)/2$. One can show something like this as an upper bound; but not I think the asymptotic. $\endgroup$ – Lucia Feb 22 '14 at 18:57
It may be worth remarking that since ${\rm gcd}(p1,\frac{p^{2}+1}{2}) = 1,$ every group of order $\frac{p(p^{2}+1)}{2}$ has a normal subgroup of index $p$ by Burnside's normal $p$complement theorem. On the other hand, no positive divisor of the odd integer $\frac{p^{2}+1}{2}$ (other than 1) can be congruent to 1 (mod $p$). Hence it is the case that any group of order $\frac{p(p^{2}+1)}{2}$ has a unique Sylow $p$subgroup, so is a direct product of a group of order $p$ and a group of order $\frac{p{^2}+1}{2}.$ Whether or not the resulting group is Abelian depends only on the structure of the latter group of order $\frac{p^{2}+1}{2},$ which most answers to date have concentrated on anyway. However, it is the case that any such group (of order $\frac{p(p^{2}+1)}{2}$ ) decomposes as a direct product of two smaller nontrivial groups. I don't know if there is a characterization of integers $n$ such that every finite group of order $n$ is decomposable.

$\begingroup$ Can we say that this subgroup of order $(p^2+1)/2$ is a characteristic subgroup? $\endgroup$ – BHZ Feb 25 '14 at 10:44

$\begingroup$ Yes, we can certainly say that. It is the unique subgroup of that order. $\endgroup$ – Geoff Robinson Feb 25 '14 at 13:29