Let $p>3$ be a prime number and $G$ be a finite group of order $2p(p^2+1)$. Is it true that always the Sylow $p$ subgroup of $G$ is a normal subgroup of $G$? As I checked by Gap it seems true. Thanks for your answer.
As $p^2+1 \equiv 2 \bmod 4$, a Sylow $2$subgroup of $G$ has order four. As $p^2+1 \equiv 2 \bmod 3$ and $p>3$, $G$ has no element of order three. It follows now that a Sylow $2$subgroup $X$ of $G$ is central in its normalizer in $G$. By Burnside's normal $p$complement theorem, $G$ contains a normal complement $N$ to $X$. It suffices to show that $N$ has a normal Sylow $p$subgroup, as such a subgroup is characteristic in $N$ and thus normal in $G$.
Let $t$ be the number of Sylow $p$subgroups of $N$. Then $t \equiv 1 \bmod p$ and $t$ divides $\frac{p^2+1}{2}$. Assume for contradiction that $t>1$, write $t=ap+1$ with $a>0$. Then $p^2+1=2b(ap+1)$ for some integer $b$. Now $2b \equiv 1 \bmod p$. So, $2b>p$ and $2b(ap+1)>p^2+p$, giving the desired contradiction.
Let $k$ be the number of $p$Sylow subgroups. Then $k$ divides $2(p^2+1)$. Let $l$ be the $2(p^2+1)/k$. $k$ is congruent to $1$ mod $p$, so $l$ is congruent to $2$ mod $p$. So $k=ap+1$ and $l=bp+2$. But $ab\leq 2$ clearly so we can divide into three cases:
$a=1,b=1$: $(p+1)(p+2) \neq 2p^2+2$ unless $p=3$.
$a=2, b=1$ $(2p+1)(p+2) \neq 2p^2+2$
$a=1,b=2$: $(p+1) (2p+2) \neq 2p^2+2$
so each case is impossible, except $a=0$ and $b=0$. If $a=0$ we are done, so we consider the case $b=0$.
The normalizer of a Sylow subgroup is a group of order $2p$, hence it is either $\mathbb Z/{2p}$ or $D_p$. This gives a division into two cases.
For another division into two cases, either this subgroup is a Frobenius complement, or the normalizers of different Sylow subgroups intersect.
This gives $2\times 2=4$ cases. However, the subgroup cannot be a Frobenius complement, because $\mathbb Z/p$ cannot act nontrivially on a group of order $p^2+1$. Since $p^2+1=2$ mod $4$, the Sylow $2$subgroups are all $\mathbb Z/2$, so the number of elements of order $2$ divides $p^2+1$, so it is not a multiple of $p$, so there is a $p$invariant element. But the group of $p$invariant elements also has order dividing $p^2+1$, and is congruent to $1$ mod $p$ (because the order of its complement is a multiple of $p$), so its order is $1$ or $p^2+1$, so the action is trivial.
We can further use this fact to eliminate the case where the group is $\mathbb Z/2p$. Let $e$ be an element shared by the normalizers of two different Sylow $p$subgroups, then it must have order $2$, and it commutes with those elements, so $Z(e)$ has at least two different Sylow $p$subgroups. The order of $Z(e)$ divides $2p(p^2+1)$, so by the same argument as the beginning, the number of Sylow subgroups in $Z(e)$ is $p^2+1$, so $Z(e)/(e)$ has order $p(p^2+1)$, so the Sylow $p$subgroup in that is a Frobenius complement. But this is a contradiction because $\mathbb Z/p$ cannot act nontrivially on a group of order $p^2+1$.
This leaves just one case, when the normalizer of a Sylow $p$subgroup is $D_p$, and different normalizers do intersect. I do not know how to eliminate this case.

2$\begingroup$ The $D_p$ case is not settled yet? $\endgroup$ – user6976 Aug 5 '13 at 6:07