This question is closely related to the restricted Burnside problem: given numbers $m$ and $p$, is the restricted Burnside group $B_0(m,p)$ finite? Every group with $m$ generators of exponent $p$ is the quotient of the Burnside group $B(m,p)=F_m/\langle w^p\rangle,$ where $F_m$ is a free group with $m$ generators, and $B_0(m,p)$ is the quotient of $B(m,p)$ by the intersection of all subgroups of finite index (which is a normal subgroup). For the case of prime exponent, A.I. Kostrikin proved that the restricted Burnside problem has affirmative solution (and Efim Zelmanov proved it in general). Thus the answer to the original question is:
A finite group $G$ has the property that all non-unit elements have the same order $p$ if and only if $p$ is prime and $G\ne 1$ is a quotient of $B_0(m,p)$ for some $m.$
For small values of $p$, even the Burnside group $B(m,p),$ which is somewhat easier to study, is known to be finite ($p=2,3$) and one may hope to get a more precise answer (for $p=2$ the group is elementary abelian 2-group of rank $m$).