Let me see if I can write down a proof that the answer is YES for most of the $A_n$-groups. The method should work for the rest of the $A_n$-groups and, indeed, all of the other groups you mention. Specifically, I'll prove

**Proposition**: If $G={\rm PSL}_n(q)$ with $n\geq 5$ and $q\geq 2$, then $\mu(G)>4$.

**Proof**: In what follows, I'll write $q=p^f$ where $p$ is a prime, $f$ a positive integer. Then $|{\rm Aut}(G)|/ |{\rm PGL}_n(q)|=2f$. Recall that a *primitive prime divisor* of $p^{df}-1$ is a prime that divides $p^{df}-1$ but not $p^k-1$ for any integer $k<df$. Zsigmondy's theorem asserts that such a prime always exists unless $df=2$ or $(p,df)=(2,6)$.

An easy argument shows that if $r$ is a primitive prime divisor of $p^{df}-1$, then $r>df$. It's written down as Lemma 2.7 of a paper of mine with Azad and Britnell. This is important because it means that primitive prime divisors of $p^{df}-1$ do not divide $2f$ provided $df>1$.

So now our job is to find five elements, $g_1,\dots, g_5\in G$, with different orbit sizes under ${\rm Aut}(G)$. Write $o(g_i)$ for the the orbit size of $g_i$. In what follows we note down what makes each $o(g_i)$ definitely different to the others.

- Let $g_1=1$. Then $o(g_1)=1$.
- Let $g_2$ be central in a Sylow $p$-subgroup of $G$. Then $o(g_i)$ is not divisible by $p$.
- Let $g_3$ be an element whose centralizer is a maximal torus in ${\rm PGL}_n(q)$ of size $\frac{q^n-1}{q-1}$. Then $o(g_3)$ is not divisible by a ppd of $q^n-1$.
- Let $g_4$ be an element whose centralizer is a maximal torus in ${\rm PGL}_n(q)$ of size $q^{n-1}-1$. Then $o(g_4)$ is not divisible by a ppd of $q^{n-1}-1$.
- Let $g_5$ be an element whose centralizer is a maximal torus in ${\rm PGL}_n(q)$ isomorphic to $(q^{n-2}-1)\times(q-1)$. Then $o(g_5)$ is not divisible by a ppd of $q^{n-2}-1$.

Some remarks:

- the existence of tori of this size can be seen directly, but is written down explicitly in the paper of Buturlakin and Grechkoseeva.
- The fact that a ppd of $q^{n-2}-1$ does not divide $q^n-1$ follows from the fact that $n\geq 5$ and the fact that $gcd(q^{n-2}-1, q^n-1)$ divides $q^2-1$. Similarly for the other pairs of ppds.
- We are implicitly using the lower bound on ppd's used above -- so that the action of $|{\rm Out}(G)|$ doesn't mess things up.
- In theory one should check what happens when $(q,n)\in\{(2,6), (2,7), (2,8)\}$ -- here Zsigmondy's theorem fails for one of the tori mentioned above and a ppd does not exist. But I've excluded $q=2$ so this doesn't arise (see next comment).
- Finally the fact that there really are elements that are centralized by these maximal tori is a fairly straightforward eigenvalue argument. The only problem arises when $q=2$ and we have the torus $(q^{n-2}-1)\times (q-1)$, hence I've excluded $q=2$ from the statement of the theorem. QED

**Extra remarks**

- Dealing with $q=2$ should be easy. For instance if $n$ is odd, then you can substitute that torus of size $(q^{n-2}-1)\times (q-1)$ with one of size $(q^{n-2}-1)\times (q+1)$ (maybe this will work for $n$ even too but I haven't checked).
- Likewise $n=2,3,4$ can be done by hand (I guess).
- The whole argument carries over to the unitary groups just by changing a few signs in the size of the tori. The other classical groups will require that you choose tori of different sizes.