There are two nonabelian groups of order $p^3$, namely, semidirect product of $\mathbb Z /p \mathbb Z \times \mathbb Z /p \mathbb Z$ by $\mathbb Z /p \mathbb Z$ and semidirect product of $\mathbb Z /p^2 \mathbb Z$ by $\mathbb Z /p \mathbb Z$. What are the automorphism groups of these groups?

2$\begingroup$ Have you tried asking GAP? $\endgroup$ – Mariano SuárezÁlvarez Jan 21 '11 at 5:11

$\begingroup$ In GAP, I tried for first group with p=3,( i.e. G=Z/3Z x Z/3Z): Z/3Z); but no output. It gives Aut(G)=432, but no structure description. So I am looking theoretically. I did following: since any automorphism of G fixes center(not necessarily pointwise), there is natural homomorphism f:Aut(G) > Aut(G/Z(G)), where Z(G)=p, and G/Z(G)=Z/pZ x Z/pZ, Aut(G/Z(G))=GL(2,p). We can show that f is surjective with kernel Z/pZ x Z/pZ. Hence Aut(G)=(p^2).GL(2,p)=(p^3).(p+1).(p1)^2. I couldn't proceed further. $\endgroup$ – Soluble Jan 21 '11 at 6:02

4$\begingroup$ It's instructive to work out such examples from scratch, but it's also important to realize that this has certainly been done before and is most likely written down in detail (somewhere). With over a century of literature on group theory including many books, it could take a while to locate such information; maybe it's quicker to do it yourself. Anyway, it's also useful to organize the possible methods such as generators and relations, linear representations, geometric realizations. GAP has its limitations for giving insight. $\endgroup$ – Jim Humphreys Feb 22 '11 at 15:32

$\begingroup$ This example I couldn't see in any Graduate Studies book; it is not also given as an exercise. Late, I found that it is an exercise in the book "Structure of Groups of Prime Power Order". When working such examples in GAP, it gives the order of Automorphism group of these groups quickly, but it is not giving information about structure. $\endgroup$ – Soluble Feb 23 '11 at 3:55

$\begingroup$ @Rahul: I deleted my answer (I think), and instead typed everything up into a PDF. You can email me and I'd be glad to send it to you. $\endgroup$ – Steve D Feb 23 '11 at 9:11
For the latter group, the answer is Bidwell & Curran's paper "The Automorphism Group of a Split Metacyclic pGroup"

2
The former group can be seen as the group of unitriangular $3 \times 3$matrices over the field with $p$ elements: $$G = \left\{ \begin{pmatrix} 1 & * & * \\ 0 & 1 & * \\ 0 & 0 & 1 \end{pmatrix} \right\} \leq SL(3,p)$$ The automorphism groups of such groups have been studied (in a much larger generality); see for instance the paper "The automorphism group of the group of unitriangular matrices over a field" by Ayan Mahalanobis (http://arxiv.org/abs/1012.5534v1).
The automorphism groups of all pgroups of order p^3 can be found at http://www.math.kth.se/~boij/kandexjobbVT11/Material/pgroups.pdf
There is a clear and more specific answer (with reference moreover!) here, despite the different question: https://math.stackexchange.com/a/18496/84625
In short $\operatorname{Aut}\left(\left(\mathbb{Z}_p \times \mathbb{Z}_p\right) \rtimes \mathbb{Z}_p) \right) \cong \operatorname{AGL}(2,p)$ while $\operatorname{Aut}\left(\mathbb{Z}_{p^2} \rtimes \mathbb{Z}_p \right) \cong \left(\mathbb{Z}_p \times \mathbb{Z}_p\right) \rtimes \operatorname{AGL}(1,p)$
where $\operatorname{AGL}(n,p)$ is the General Affine Group .