Do you know what such a representation theoretic argument would look like? Thanks for your list. It still seems a bit of a mystery where conductor 50 comes from, though. Then again, there might not be a nice explanation, in which case this might be a bad question.

I must confess I got the information on the conductors being 50 from the Antwerp volume you cite. Apparently I overestimated the 2-descent, and it's very reassuring to know one can always just do that, thanks! I should probably rephrase the bit about "reasonable computations"... That said, I would still be very excited if there were a good answer to either of the questions above.

Since "most" groups with bounded order are $p$-groups, this deals with quite many of them. :-) Can anything be said about other families? For example, I think one has all the dihedral groups. Is there a positive answer for other families of non-abelian groups?

Very nice answer, the point you make on the quotients is quite final. Thanks!

Accepted for the simplest criterion. I never realised this nice fact about odd Sylow subgroups of $GL_2(q)$! As an example, it follows from your answer that the only subgroups $A_n$ have $n\leq 5$, where we note that $A_5 \times C_3 \cong GL_2(4)$. Your arguments exclude the bigger ones, since $q$ would have to be even, resulting in abelian Sylow $2$-subgroups. This seems to suggest a general phenomenon, where only "small" groups will arise. In general the Sylows will be unpredictable and probably non-abelian, which gives a notion of "almost never" as in the question. Thanks!