There is a description of the automorphism groups of groups of order $p^4$ in the Thesis by [B. Girnat: "Klassifikation der Gruppen bis zur Ordnung $p^5$." Staatsexamensarbeit, TU Braunschweig, 2003][1], whose advisor was B. Eick.
The description is given in terms of the action on a set of generators of the respective group.
If I may add a personal note, I have been searching for this information for a long time, and I found incredible that such a basic and old topic had no treatment whatsoever published in any form on the web. But I was assuming that anything interesting can be found in English (either written or translated into it). This assumption has been proven wrong, and I state this here as a warning for the newbies like me!  


  [1]: http://www.math.uni-muenster.de/u/bgirn_01/texte/pgruppen.pdf