There are at most 2 groups (up to isomorphism) of order $n$, and there is not a field of order $n$, if and only if $n$ is the product of two distinct primes.


This doesn't cover "prime powers", but at least it's nontrivial.