Let me expand my comment to  a partial answer of "why sparsity?". It is impossible for $|{\rm PSL}(2,p^{m})|$ to have four or fewer prime factors when 
$p$ is an odd prime which is neither Fermat nor Mersenne and $m >1.$ 

If $m$ is itself even, then $p^{2m}-1$ is divisible by $p^{4}-1$. It is easy to check that $\frac{p-1}{2}, \frac{p+1}{2}$ and $\frac{p^{2}+1}{2}$ are pairwise coprime. Also $\frac{p^{2}+1}{2}$ is odd, while one of $\frac{p \pm 1}{2}$ is even, and the other is odd. Hence if the odd prime $p$ is neither a Fermat prime nor a Mersenne prime, then $p^{4}-1$ is even, and has at least three different odd prime factors, so has $4$ or more different prime factors.

If $m$ is odd, it suffices to prove that $p^{2q}-1$ has four or more prime factors whenever $q$ is an odd prime and $p$ is an odd prime which is neither Fermat nor Mersenne. But by Zsygmondy's Lemma, $\frac{p^{q}-1}{p-1}$ has a prime factor $r$ with $r \equiv 1$ (mod $q$). Then $r$ is certainly odd.
If $r$ divides $p \pm 1$,  then $ 1 + p + \ldots +p^{q-1}  \equiv q$ or $1$  (mod $r$), a contradiction.

Hence $p^{2q}-1$ has at least four prime factors: $2,r$ and at least one odd prime factor of each of $\frac{p-1}{2}$ and $\frac{p+1}{2}$ ( these being different from each other,odd, and different from $r$).