For the case $n = p(p-1)(p+1)$ for $p >3$ a prime, a theorem of M. Herzog (to be found in the article "Finite groups with a large cyclic Sylow subgroup" (MSN) in the conference proceedings "Finite Simple Groups" (Oxford, editors M. Powell and G. Higman, published 1971 by Academic Press (MSN))) answers your question postively. The Theorem of Herzog uses a 1958 Theorem of Brauer and Reynolds ("On a problem of E. Artin", Annals of Mathematics, 68 (MSN)) which probably suffices to cover the cases you are interested in.
Note that if $n = p(p-1)(p+1)$ with $p >3$ a prime, and $G$ is a perfect group of order $n$, then $G$ is not $p$-solvable (for if $G$ is $p$-solvable of order $n$, then $G$ either has a factor group of order $p$ or a non-trivial cyclic factor group of order dividing $p-1$).
The Theorem of Herzog/Brauer–Reynolds proves that a finite group $G$ of order less than $p^{3}$ (for a prime $p >3$) which is not $p$-solvable is one of the following groups: $\operatorname{PSL}(2,p)$, $\operatorname{PSL}(2,p-1)$ (for $p >5$ a Fermat prime), $\operatorname{SL}(2,p)$, $\operatorname{PGL}(2,p)$, $\operatorname{PSL}(2,p) \times \mathbb{Z}/2\mathbb{Z}$.
The only perfect group of order $n$ on the list is ${\rm SL}(2.p)$ and the only perfect group of order $\frac{n}{2}$ is ${\rm PSL}(2,p)$, showing that ${\rm SL}(2,p)$ and ${\rm PSL}(2,p)$ ( for $p >3$ a prime) are the unique perfect groups of their orders.