I believe that for $p\ge5$, these are the direct products of cyclic groups of pairwise distinct orders, for if $C_{p^e}\times C_{p^e}$ is a direct factor of $G$, then $GL(2,p)$ is a homomorphic image of a subgroup of $\text{Aut}(G)$.A=\text{Aut}(G)$.
Similarly for $p=2$ or $3$, I expect the groups $G$ you are looking for are those where no order in the direct product appears more than $2$ times.
In order to show that these groups indeed have a solvable automorphism group, you probably may apply induction: Let $N$ be the subgroup of $G$ generated by the $p$-th powers in $G$. Then $N$ has the same shape as above (with the order of each direct factor divided by $p$). So $\text{Aut}(N)$ is solvable by induction. As $C_G(N)$ C_A(N)$ is the kernel of the restriction of $G$ A$ to $N$, all what remain to show is that $C_G(N)$ C_A(N)$ is solvable too.
Alternatively, one could try to use the induction hypothesis for $G/K$, where $K$ is generated by the elements of order $p$. Note that $N$ is (non-canonically) isomorphic to $G/K$.
