Let me expand my comment to  a partial answer of "why sparsity?". It is impossible for $|{\rm PSL}(2,p^{m})|$ to have four or fewer prime factors when 
$p$ is an odd prime which is neither Fermat nor Mersenne and $m > 1$ is an integer. In fact, if $m$ is not a power of $2$, we can do somewhat better.

If $m$ is itself even, then $p^{2m}-1$ is divisible by $p^{4}-1$. It is easy to check that $\frac{p-1}{2}, \frac{p+1}{2}$ and $\frac{p^{2}+1}{2}$ are pairwise coprime. Also $\frac{p^{2}+1}{2}$ is odd, while one of $\frac{p \pm 1}{2}$ is even, and the other is odd. Hence if the odd prime $p$ is neither a Fermat prime nor a Mersenne prime, then $p^{4}-1$ is even, and has at least three different odd prime factors, so has $4$ or more different prime factors.

If $m$ is not a power of $2$, we claim that $p^{2r}-1$ has four or more prime factors whenever $r$ is an odd prime and $p >3$ is an odd prime. 

For $\frac{p-1}{2}, \frac{p+1}{2}, \frac{p^{r}-1}{p-1}$ and $\frac{p^{r}+1}{p+1}$ are four integers, of which 
exactly one is even. If $p >3$ is a prime, then the product
$\frac{p+1}{2}\frac{p-1}{2}$ has at least two prime factors, since $p-1$ and $p+1$ can't both be powers of $2$ when $p>3$.

Note that both of $\frac{p^{r} \pm 1}{p \pm 1}$ are odd, and they are coprime to each other because ${\rm gcd}(p^{r}-1,p^{r}+1)$ divides $2$. 

If $s$ is a prime which divides both $p-1$ and $\frac{p^{r}-1}{p-1}$, then we see easily that $\frac{p^{r}-1}{p-1}$ is congruent to $r$ mod $s$, so we must have $r = s.$ In that case, we have ${\rm gcd}(p-1, \frac{p^{r}-1}{p-1}) = r$, and since $\frac{p^{r}-1}{p-1}$ is certainly greater than $r$, we see that $\frac{p^{r}-1}{p-1}$ has a (necessarily odd) prime factor $s$ which does not divide $p-1.$

Notice that $s$ does not divide $p^{r}+1$, for otherwise $s$ must divide $p+1$, and then $\frac{p^{r}-1}{p-1}$ is congruent to $-1$ (mod $s$, a contradiction.

Likewise, we find that $\frac{p^{r}+1}{p+1}$ has a (necessarily odd) prime factor $t$ which does not divide $p^{2}-1.$ We have to be a little more careful in noting that $1-p + p^{2} - \ldots - p^{r-2} + p^{r-1}$ is 1+ (a sum of $\frac{r-1}{2}$ pairs, each at least $p-1 \geq 4$), so that $\frac{p^{r}+1}{p+1} \neq r$.

In conclusion, we see that the odd prime  $p$ is neither a Fermat nor a Mersenne prime, then $|{\rm PSL}(2,p^{2m})|$ has at least 5 prime divisors for each integer $m >0$, while if $p$ is any prime greater than $3$, then  $|{\rm PSL}(2,p^{m})|$ has at least $5$ prime divisors whenever $m > 1$ is an odd integer.