If there are exactly two groups (up to isomorphism) of order $n$, then $n$ is the product of two primes.

This isn't if and only if, it doesn't capture "distinct", and it doesn't cover "prime powers", but at least it's nontrivial.