Yes.:
for every prime $p\ge 7$, there's a finite group of exponent $p$ whose automorphism group is a $p$-group.
StartInitial answer (Jan 18' 2016) Start from any (finite-dimensional) complex nilpotent Lie algebra $\mathfrak{g}$ that is defined over $\mathbf{Q}$ and has a unipotent automorphism group (see e.g. Luks' Lie algebra top of p14 in (Ancochea Campoamor survey) for a 16-dimensional example), see also first edit (Jan 19' 16) below.
Edit: (Jan 19' 2016) I have checked that the 7-dimensional (with nilpotency length 6) Lie algebra given page 17 line 8 in (Ancochea Campoamor survey), viewed with coefficients in an arbitrary field, has only unipotent automorphisms. If one takes it in the field on $p$ elements for any prime $p\ge 7$, the Baker-Campbell-Hausdorff (which realizes Malcev's correspondence) provides a group of order $p^7$ whose automorphism group is a $p$-group (thus of odd order).
Edit: (Jan 25' 2018) Thinking a little about the remaining cases $p=3,5$, they seem different and deserve separate additional comments:
first recall by above remark that for any odd $p$, the nilpotency length of any example should be $\ge 3$.
- for $p=5$, Malcev correspondence is valid in nilpotency length $\le 4$. Therefore if one searches among nilpotent rational Lie algebras of nilpotency length in $\{3,4\}$ and unipotent automorphism groups, one can reasonably hope to find one that reduces modulo 5 and for which the reduction modulo 5 has no extra automorphisms.
- $p=3$ is quite different, because exponent 3 implies nilpotency length $\le 3$ (in contrast, by Razmyslov's theorem, for all $p>3$ there are finite groups of exponent $p$ and arbitrary large solvability length). So in this case, Lie algebras are of no help. Finding a finite group of exponent 3 whose automorphism group is a 3-group (or of odd order) sounds a quite different question.