This question is closely related to the restricted Burnside problem: given numbers $m$ and $p$, is the restricted Burnside group $B_0(m,p)$ finite? Every group with $m$ generators of exponent $p$ is the quotient of the Burnside group $B(m,p)=F_m/\langle w^p\rangle,$ where $F_m$ is a free group with $m$ generators, and $B_0(m,p)$ is the quotient of $B(m,p)$ by the intersection of all subgroups of finite index (which is a normal subgroup). For the case of *prime* exponent, A.I. Kostrikin proved that the restricted Burnside problem has affirmative solution (and Efim Zelmanov proved it in general). Thus the answer to the original question is:

A finite group $G$ has the property that all non-unit elements have the same order $p$ if and only if $p$ is prime and $G\ne 1$ is a quotient of $B_0(m,p)$ for some $m.$

For small values of $p$, even the Burnside group $B(m,p),$ which is somewhat easier to study, is known to be finite ($p=2,3$) and one may hope to get a more precise answer (for $p=2$ the group is elementary abelian 2-group of rank $m$).