There is a related paper for the general case, where all elements (nontrivial) has prime power order. Although, this paper does not answer your question in generally, but it has some good techniques for attacking to such problem. The paper name is:

"Classification of Finite Groups with all Elements of Prime Order" by "Marian Deaconescu".

In this paper, he studied the variant of above question and obtained some results as follows:

Let $\mathcal{P}$ be the class of the finite groups having all (nontrivial) elements of prime order. Let $G$ be a $\mathcal{P}$-group. Then one of the following cases occurs:

$\text{I}.$ $G$ is a $p$-group of exponent $p$.

$\text{II}.$ $(a)$ $|G|=p^aq$, $3\leq p<q$, $a\geq 3$, $|F(G)|=p^{a-1}$, $|G:G'|=p$.

$(b)$ $|G|=p^aq$, $3\leq q<p$, $a\geq 1$, $|F(G)|=|G'|=p^a$.

$(c)$ $|G|=2^ap$, $p\geq 3$, $a\geq 2$, $|F(G)|=|G'|=2^a$.

$(d)$ $|G|=2p^a$, $p\geq 3$, $a\geq 1$, $|F(G)|=|G'|=p^a$ and $F(G)$ is elementary abelian.

$\text{III}.$ $G\cong A_5$.