From MathReviews:

MR0367059 (51 #3301)
Jonah, D.; Konvisser, M.
Some non-abelian $p$-groups with abelian automorphism groups.
Arch. Math. (Basel) 26 (1975), 131--133.

This paper exhibits, for each prime $p$, $p+1$ nonisomorphic groups of order $p^8$ with elementary abelian automorphism group of order $p^{16}$. All of these groups have elementary abelian and isomorphic commutator subgroups and commutator quotient groups, and they are nilpotent of class two. All their automorphisms are central. With the methods of the reviewer and Liebeck one could also construct other such groups, but the orders would be much larger.

FYI, I found this via a google search.

The first to construct such a group (of order $64 = 2^6$) was G.A. Miller* in 1913. If you know something about this early American group theorist (he studied groups of order 2, then groups of order 3, then...and he was good at it, and wrote hundreds of papers!), this is not so surprising. I found a nice treatment of "Miller groups" in Section 8 of

http://arxiv.org/PS_cache/math/pdf/0602/0602282v3.pdf

(*): The wikipedia page seems a little harsh. As the present example shows, he was a very clever guy.