If there are exactly two groups (up to isomorphism) of order $n$, then $n$ is the product of two primes.

This isn't if and only if, it doesn't capture "distinct", and it doesn't cover "prime powers", but at least it's nontrivial.

There are at most 2 groups (up to isomorphism) of order $n$, and there is not a field of order $n$, if and only if $n$ is the product of two distinct primes.

This doesn't cover "prime powers", but at least it's nontrivial.